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EXECUTIVE SUMMARY 


As part of its ongoing efforts to raise the academic achievement of children from low- 
income families in Kansas City, Missouri, the Ewing Marion Kauffman Foundation founded the 
Ewing Marion Kauffman School in fall 2011. The Kauffman School’s mission is “to prepare 
students to excel academically, graduate from college, and apply their unique talents in the world 
to create economically independent and personally fulfilling lives” (Ewing Marion Kauffman 
School 2013). 

As a public charter school, the Kauffman School is tuition-free and serves students living in 
Kansas City. In the 2014-15 school year, the Kauffman School enrolled 561 students in grades 
5-8; approximately 85 percent of them were low-income and 89 percent were black or Hispanic. 

The Kauffman School has ambitious goals for its students, including accelerated learning, 
high attendance levels, and exemplary behavior. In this report, we summarize information about 
the impact of the Kauffman School on student achievement, attendance, and rate of suspensions. 

Data and methods. Data used for this report came from the Missouri state department of 
education and included: student achievement on the Missouri Assessment Program (MAP) 
exams, attendance, suspensions, and demographic characteristics of the students. To estimate the 
impact of the Kauffman School on its students, we identified a group of similar students 
attending other Kansas City schools and then compared the two on key impact measures. More 
details on our analytic approach are provided in the full report. 

Main findings. Our findings indicate that in the 201 1-12 through 2014-15 school years, the 
Kauffman School had positive, statistically significant, and educationally meaningful impacts on 
student achievement growth in mathematics, ELA, and science beyond the growth achieved by 
students in other Kansas City public schools. 

In Table ES.l below, results are reported separately for students one, two, three, and four 
years after entering the Kauffman School in 5th grade. In every subject and year examined, the 
Kauffman School’s impact on test scores is positive and statistically significant, which indicates 
that it is outperforming other Kansas City schools serving similar students. 
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Table ES.1. Impact of Kauffman School on MAP test scores (citywide 
comparison group) 1 



Mathematics 

ELA 

Science 

Impact one year after enrollment (5th grade) 

0.22** 

0.23** 

0.46** 

Impact two years after enrollment (6th grade) 

0.35** 

0.18** 

n.a. 

Impact three years after enrollment (7th grade) 

0.68** 

0.52** 

n.a. 

Impact four years after enrollment (8th grade) 

0.96** a 

0.53** 

0.66** 


Notes: There are no two- or three-year estimates for science, because the state does not have a science test for 
6th or 7th graders. 

‘‘Significantly different from zero at the 1 percent level. 

a The four-year mathematics impact is based in part on imputed outcome data. See footnote 2 for details. 

ELA = English language arts; n.a. = not applicable. 

The effect-size units reported in Table ES.l are useful but not very intuitive. To translate the 
results into units that are more readily interpretable, we turn the effect sizes into years of 
additional learning growth through a commonly used conversion method for effect sizes (Bloom 
et al. 2008). Figure ES.l below displays these impact estimates converted to years of additional 
learning growth for Kauffman students. 

Impacts on mathematics achievement growth. The estimated impact of the Kauffman 
School on student achievement in mathematics is substantial. 2 The magnitude of the effect size 
translates into approximately 1.61 additional years of learning growth three years after 
enrollment. Before entering the Kauffman School (that is, in 4th grade), the average Kauffman 
student is at the 39th percentile in the state in mathematics. These results suggest the average 
Kauffman student would move to the 65th percentile three years after enrollment at the 
Kauffman School. Moreover, the three-year mathematics effect is equal to approximately 84 
percent of the test score achievement gap between black and white students in 7th grade in 
Kansas City. 


1 The impact of the Kauffman School on student achievement growth is reported in “effect size” units 
(fractions of standard deviations of student test scores) that are commonly used in education studies and that allow 
comparisons to other studies. We measure the average effect that attending the Kauffman School has on student test 
score growth beyond what students would have achieved if they had attended other Kansas City public schools. A 
positive effect size means that test score growth is higher for Kauffman students relative to comparison students, and 
vice versa. 

O 

The four-year mathematics impact should be interpreted with caution, because not all of the students in the 
matched comparison group took the 8th-grade MAP: students who were taking Algebra I in 8th grade took a 
different test. To deal with this problem, we imputed missing 8th-grade mathematics MAP scores for 8th-grade 
students taking Algebra I. See section III.A for details on our imputation process. This imputation inherently adds 
some uncertainty to the exact size of the four-year mathematics impact, but we believe this approach provides a 
reasonable approximation. In this report we focus primarily on the three-year impact estimates for mathematics 
when discussing the magnitude of the effect of the Kauffman School on student achievement. To simplify the 
comparisons of these impacts with results from other studies, we will focus on the three-year ELA impact estimates 
as well. The three-year impact estimates in this report also have the advantage of being based on two cohorts of 
students rather than one. 
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Figure ES.1. Kauffman School estimates of additional years of learning 
growth on MAP exams 
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Notes: The additional growth for all impact estimates is significantly different from zero. 
a The four-year mathematics impact is based in part on imputed outcome data. See footnote 2 for details. 

Impacts on ELA achievement growth. The effect size in ELA is substantial as well; the 
magnitude translates into approximately 1 .64 additional years of learning growth by the end of 
the third year after enrolling. The average Kauffman student moves from the 39th percentile in 
the state in ELA before entering the school to the 60th percentile after three years. This effect is 
approximately 84 percent of the ELA test score achievement gap between black and white 
students in 7th grade in Kansas City. 

Impacts on science achievement growth. The impact of the Kauffman School in science is 
also large. Four years after enrolling in the Kauffman School, students achieved approximately 
2.18 additional years of learning in science compared with students at other Kansas City schools. 
This is equivalent to approximately 58 percent of the local science test score achievement gap 
between black and white students in 8th grade. However, the science impact estimate should be 
interpreted with caution, because there was no 4th-grade science exam that could be used in the 
analysis; instead, 4th-grade ELA and mathematics scores were used as baseline controls. 

Comparison to other charter schools. The Kauffman School’s achievement impacts in 
mathematics and ELA three years after enrollment are larger than the average effects observed 
for other highly successful charter school programs (Figure ES.2), including: the average Boston 
charter school analyzed by Abdulkadiroglu et al. (2009), the average Knowledge Is Power 
Program (KIPP) middle school studied by Tuttle et al. (2013), and the average New York City 
charter school analyzed by Hoxby et al. (2009) (although some individual schools in those 
studies achieved higher impacts than the Kauffman School). The estimated two-year impacts of 


2.42 a 


2.18 
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the Kauffman School are smaller, but within the range of these highly successful charter school 
programs. See section III.C for further details. 

Moreover, the Kauffman School is strongly outperforming broader samples of charter 
schools nationwide. The effects of the Kauffman School are substantially larger than those of the 
average oversubscribed charter school serving a large fraction of low-income students analyzed 
by Gleason et al. (2010), the average urban charter school in the 41 regions analyzed by the 
Center for Research on Education Outcomes (CREDO 2015), and the average school in a 
nationwide group of charter school management organizations (CMOs) studied by Furgeson et 
al. (2012). 

Figure ES.2. Charter school three-year impact estimates from various studies 
represented as years of additional learning growth 3 



lotteries study 


h Mathematics a Reading/English language arts 

Notes: Figure ES.2 contains three-year effect size estimates converted to years of additional learning growth. 

The impact of charter schools on science achievement is less widely reported because 
science tests are administered in fewer grades in most states. The Kauffman School, with an 
estimated four-year effect size of 0.66, is performing well compared to KIPP middle schools 
which are estimated to have a cumulative average impact of 0.33 standard deviations in science 
for students three to four years after enrollment (Tuttle et al. 2013). 


Effect size estimates for the average Boston charter school as reported in Abdulkadiroglu et al. (2009), for the 
average KIPP charter school analyzed by Tuttle et al. (2013), the average New York City charter school in grades 4 
through 8 as reported in Hoxby et al. (2009), the average charter school with a lottery admission process serving a 
large fraction of low-income students analyzed by Gleason et al. (2010), the average urban charter school in the 41 
regions analyzed by the Center for Research on Education Outcomes (CREDO 2015), and the average school in the 
charter school management organizations (CMOs) studied by Furgeson et al. (2012). See Section III.C for further 
details. 
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Alternate comparison groups in Kansas City. The main findings summarized here are a 
result of comparing Kauffman students with a matched comparison group of students from all 
public schools in Kansas City. We also compared Kauffman students with two subgroups of 
children — (1) similar students attending district-operated schools in Kansas City and (2) similar 
students attending other Kansas City charter schools. The estimated effect sizes are generally 
higher when the Kauffman School is compared only with district-operated schools and lower 
when compared only with other charter schools. All the effect size estimates for both comparison 
groups are positive and significant, indicating that the Kauffman School is outperforming the 
average charter school and the average district school in Kansas City in all three tested subjects. 

Changes in effectiveness of the Kauffman School over time. We analyzed whether the 
impact of the Kauffman School on student achievement changed during the first four years of 
operation by examining changes over time. The Kauffman School’s estimated impacts in year 
four (2014-15) were higher in almost all grades and subjects compared to previous years. Even 
though the Kauffman School was producing significant achievement impacts in earlier years, it 
appears to have substantially accelerated its achievement impacts in the most recent year. The 
significant increase may be related to the change in standards tested on the state standardized 
exams to align with the Common Core State Standards. The Kauffman School had already 
changed its curricula to align with the Common Core standards prior to year four, which likely 
positioned students to perform well on the new state tests. It is also possible that the increased 
impacts in year four were related to other factors, such as improved teacher or school 
effectiveness. 

State test proficiency goal. One of the goals of the Kauffman School is for at least 75 
percent of students enrolled for three consecutive years to score proficient or advanced on “all” 
subjects of the state test (MAP). According to school staff, the original intent of the goal is that 
75 percent of students would achieve proficiency on each state test administered to its students. 
This is an ambitious goal, as only 39 percent of incoming 5th-grade students in the 201 1-12 and 
2012-13 school years had achieved at the proficient or advanced levels in mathematics and 36 
percent in ELA. Among students who were enrolled for three consecutive years, 68 percent 
achieved proficient or advanced in mathematics and 67 percent in ELA in 2014-15. In addition, 
57 percent of students scored proficient or better on both exams. Of note, the first cohort of 
students achieved 75 percent proficiency on each state test by the end of their fourth year, with 
81 percent of students scoring proficient or better in mathematics, 76 percent in ELA, and 78 
percent in science. Overall, 64 percent of students reached proficiency on all three exams. 

Attendance and suspensions. The Kauffman School had a positive and significant impact 
on student attendance during the 2014-15 school year, with attendance rates 0.9 percentage 
points higher than those of comparison students. The results varied across grades, with positive 
and significant impacts for 5th- and 6th-grade students and impacts that were not significantly 
different from zero for 7th- and 8th-grade students. 

During 2014-15, Kauffman students were 9 percentage points more likely to receive at least 
one suspension relative to comparison students. The higher overall suspension rate was driven 
mainly by higher rates of in-school suspensions for 5th and 8th graders. Of note, the average 
number of days suspended among students who receive at least one suspension is not 
significantly different between Kauffman and matched comparison students in any of the grades 
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examined. In addition, among Kauffman students who received at least one suspension during 
2014-15, the average number of days missed was less than 3. Students at the Kauffman School 
receive the equivalent of approximately 5 weeks of additional schooling per year relative to other 
public school students in Kansas City. The average number of days suspended is therefore small 
relative to the increased instructional time offered. 

Conclusions. The Kauffman School has ambitious goals for its students: accelerated 
learning and high levels of attendance. An analysis of data from the Kauffman School’s first four 
years shows significant positive impacts on student’s academic achievement growth. While the 
Kauffman School did not meet its goal of 75 percent of students scoring proficient or advanced 
on the MAP exams after three years, its first cohort of students did reach that mark at the end of 
their fourth year (81 percent proficient in mathematics, 76 percent in ELA, and 78 percent in 
science). During 2014-15, students at the Kauffman School had average attendance rates 0.9 
percentage points higher than those of comparison students. Though the Kauffman School 
suspended students at a statistically significantly higher rate than other schools in Kansas City, 
there was a substantial decrease in the 2014-15 suspension rate relative to the previous year. The 
average number of days missed due to suspensions was also small relative to the additional 
instructional time received by Kauffman students (less than 3 days missed compared to 
approximately 5 weeks of additional schooling). 
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I. INTRODUCTION 


A. Background about the Kauffman School 

For many years, the Kauffman Foundation has focused on improving education for children 
in Kansas City. Before opening the Kauffman School, the Kauffman Foundation operated several 
programs that addressed the challenges faced in urban education, such as Project Early (an early 
childhood program), Project Choice (a high school dropout prevention program), and the 
Kauffman Scholars program (an after-school college preparatory program). The success of these 
programs led Foundation leaders to consider the impact they might have on students in Kansas 
City if they established a charter school. In March 2009, the Foundation established its school 
design team, composed of Foundation education experts and the founding executive director of 
the Missouri Charter Public School Association. This team engaged in a three-step process of 
exploration and decision making before establishing the Kauffman School. 

Step 1. Analyzing Kansas City’s educational landscape. From a review of Kansas City 
assessment data, the school design team learned that during the 2008-2009 school year, charter 
school enrollment accounted for one-third of all public school enrollment in Kansas City (North 
2009), and that, among Kansas City’s charter and non-charter schools, only 16 percent of the 
middle schools and 7 percent of the high schools had at least 50 percent of students reaching 
proficient or better on statewide mathematics assessments in 2009 (Richardson 2009). 

From the Foundation’s perspective, these data suggested that Kansas City families had a 
desire for alternatives to the city’s regular public schools, and that current charter and non- 
charter public schools were struggling to help students achieve. In light of students’ low 
academic performance, the Foundation determined that 5th grade was the optimal grade for 
students to enter its charter school, which provided have ample time to prepare struggling 
students for a college preparatory program that would begin in 9th grade. 

Step 2. Selecting a location. The Foundation intended that the Kauffman School serve 
Kansas City’s low-income families. From a review of demographic data on Kansas City, the 
school design team learned that most of the city’s low-income population lives in the eastern part 
of the city, yet the majority of the city’s 23 charter schools were located in the western part. In 
response, the Foundation selected a site in the eastern part of the city, so that the Kauffman 
School would be near its target students. Using data on household income by zip code, the design 
team identified five sections of the city with high concentrations of low-income families. 
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Students living within these five (since expanded to six) zip codes are given first preference for 
enrollment. 4 

In August 2013, the Kauffman School moved to its permanent location, in the same section 
of the city. The campus has three buildings: a middle school, a high school, and a gymnasium- 
cafeteria-commons. The high school, which was built last, opened in August 2014, so all classes 
were held in the middle school building during year 3. Design elements of the new buildings 
reflect the Kauffman School’s key values and activities. For example, the new buildings have 
interior windows to facilitate classroom observations, a central feature of the Kauffman School’s 
professional development model. According to the Kauffman School’s website, the interior 
windows create “an environment that is transparent” and encourage “staff, faculty, parents, and 
visitors to observe classroom instruction as they walk through the building” (Kauffman School 
2013). The Kauffman School also offers teacher workrooms and community spaces for small- 
and large-group meetings, such as the weekly professional development meetings and 
community events. 

Step 3. Identifying best practices. Before the school opened, the design team made 
extensive efforts to learn about the best practices of successful charter schools, a process the 
team described as the “year of learning.” The team reviewed research on charter schools and 
visited successful charter schools in New York, Massachusetts, Illinois, and Wisconsin to learn 
more about the variables that contributed to the success of those schools. 

B. Characteristics of the Kauffman School 

The Kauffman School enrolled its first class of 5th graders (about 100 students) in fall 2011 
and added a second class of 5th-graders (about 100 students) in fall 2012. In fall 2013, a third 
class of 5th graders joined the Kauffman School (about 200 students). With the opening of its 
new building, the Kauffman School had sufficient capacity to double the size of the cohort 
entering in 2013. Each year, the Kauffman School will add a new 5th-grade class of about 200 
students, ultimately resulting in a fully enrolled middle school and high school (grades 5 through 
12 ). 


The hallmarks of the Kauffman School include: (1) ambitious academic goals, (2) high 
attendance and behavioral expectations, (3) extended school day and year, (4) increased time for 
mathematics and reading instruction, (5) intensive data-driven decision making, (6) extensive 
teacher professional development, and (7) well-established cultural norms. We discuss each of 
these next. 

1. Ambitious academic goals. The Kauffman School expects its students to excel 
academically and achieve at least 1.25 years of growth in mathematics, science, and 
reading each year. These goals are discussed regularly by school administrators and 


4 The School also offers bus transportation for students who live more than one mile away, thereby providing 
access to the School to students of need across the city. During the School’s second year of operation, the 
Foundation identified an additional zip code with a high concentration of low-income students and offered first 
preference for enrollment to students living in that section of Kansas City as well. 
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staff, teachers, students, and parents. In addition, daily homework, referred to as “life 
work,” is mandatory, and students serve detention if they do not turn in their assignments. 

2. High attendance and behavioral expectations. The Kauffman School has high goals 
for student attendance (95 percent average daily attendance) and behavior (full 
observance of school policies and procedures). As a guide for student behavior, teachers 
implement the SLANT method (Sit up, Listen, Ask and answer questions, Nod your 
head, Track the speaker) that was developed by the Knowledge Is Power Program 
(KIPP). Students receive merits for positive behavior and demerits for negative 
behavior, and may serve detention (in or out of school) depending on the number of 
demerits they earn. The dean of students oversees the implementation of the Kauffman 
School’s behavioral policies. Before the start of each school year, the Kauffman School 
holds an all-parent meeting to orient incoming students and their families to its high 
behavioral expectations. 

3. Extended school day and year. With students coming from Kansas City’s low- 
performing schools, the design team anticipated that many of them would enter school 
performing below grade level and would need more instructional time to catch up 
academically and be ready for the Kauffman School’s college preparatory program. 
Thus, the Kauffman School operates an extended school day and year to provide 
students with more instructional time. 

During the Kauffman School’s first year, students received 37.8 hours of instruction per 
week (Richardson 2009). In Year 2, the weekly instructional time was shortened to 36.5 
hours in response to feedback from students, parents, teachers and staff (Gentile et al. 
2013). In Year 3, the time was reduced slightly from Year 2, to 36.2 hours. Weekly 
instructional time was similar in Year 4: 36.3 hours. With the extended school day and 
year, Kauffman students receive approximately 5 additional weeks of schooling 
compared to traditional public school students in Kansas City. 

4. Increased mathematics and reading instructional time. Every day in Year 1, 
Kauffman students attended a double period of mathematics (104 minutes), a nonfiction 
reading class (50 minutes), a writing class (50 minutes), and an extended period of 
literature (80 minutes). During Year 2, students continued to have a double period of 
mathematics and their nonfiction reading and writing classes. Their literature class 
focused on guided reading instruction and was shortened to one period of 50 minutes. 
During Year 3, students continued to receive a double period of mathematics and 50 
minutes of reading instruction. A textual analysis course was added in Year 3. The 
writing class was absorbed into the textual analysis class, and students received a double 
period of textual analysis. In Year 4 students continued to have a double period of 
mathematics (100 minutes) and three periods (150 minutes) of instruction related to 
English language arts (ELA) and reading. In Years 2, 3, and 4, all students attended a 
daily instructional support class in which they received tutoring and special instruction. 
Struggling students received additional instruction and practice in any subjects they 
needed help with, and high-performing students received advanced instruction. 

5. Intensive data-driven decision making. With its strong emphasis on results, the 
Kauffman School utilizes a large assessment portfolio so that teachers and 
administrators can make data-driven decisions about how to adapt instruction to best 
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meet students’ needs. In addition to teacher-developed “exit tickets,” 5 quizzes, and tests 
to measure understanding and academic progress, the Kauffman School’s assessment 
portfolio includes the following: 

• Achievement Network (ANet) assessments in mathematics and reading, revised by 
Kauffman School teachers to be consistent with Missouri State Standards, administered 
every six weeks 6 

• Strategic Teaching and Evaluation of Progress (STEP) assessments to measure students’ 
reading growth, administered six times per year 

• Northwest Evaluation Association (NWEA) assessments in mathematics, reading, and 
science, administered twice a year 

• Missouri Assessment Program (MAP) standardized tests in mathematics, ELA, and 
science, administered annually by the State of Missouri 

6. Extensive teacher professional development. The Kauffman School places a significant 
emphasis on teachers’ professional development, with teachers experiencing (1) weekly 
observations and feedback from administrators; (2) weekly individual coaching sessions; 
and (3) group-based professional development sessions every Friday afternoon, focused on 
various topics related to curriculum, instruction, and assessment (Gentile et al. 2014). 

7. Well-established cultural norms. The Kauffman School takes an intentional approach to 
establishing a culture of shared values, expectations, and norms, epitomized by its motto: 
“Creating College Graduates.” Continuous efforts are made to communicate explicitly — to 
all school staff, students, and families — the values, expectations, and norms. 


5 Exit tickets are short questions or tasks that students complete at the end of the class period. These enable 
teachers to track the progress of their students’ understanding of the course material on a regular basis. 

6 When the Kauffman School started using the ANet exams, it was the only network participant in the state of 
Missouri and the exams received by the school were aligned to the state standards of Tennessee. School staff 
therefore re-wrote a large portion of the questions in Year 1 to better align the exams with Missouri standards, and 
continued to update the exams in subsequent years. 
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II. METHODOLOGY AND DATA 


A. Methodology 

Comparing Kauffman students to students from other Kansas City schools. Because all 
Kauffman students have chosen to enroll in the Kauffman School, they might differ from other 
Kansas City students in important ways. Measuring the effect of the Kauffman School on student 
achievement requires identifying a comparison group of Kansas City students who, as of 4th 
grade (before the Kauffman School’s 5th-grade entry year), are similar to the students about to 
enter Kauffman School. Otherwise, any difference we find in later student outcomes might not 
really be due to the effect of the Kauffman School. 

To guarantee that the comparison group is similar, the gold standard research design would 
involve conducting a lottery wherein some of the students who apply to the Kauffman School are 
randomly selected to attend and others are randomly denied admittance. The achievement of 
these two randomly established groups could then be fairly compared (based on the assumption 
that any naturally occurring differences among students would be randomly distributed between 
the two groups). However, the Kauffman School was not sufficiently oversubscribed during its 
first four years of operation for this approach to be used. Instead, we employed the next-best 
approach: using data from students across Kansas City to identify a matched comparison group 
of students who were similar to Kauffman students at the time they were in the 4th grade. 

To construct a comparison group of students, we implemented a propensity-score matching 
procedure. Students attending other schools in Kansas City were matched to Kauffman students 
based on characteristics such as prior test scores, prior attendance, prior suspensions, and 
demographic characteristics. This approach is commonly used as an alternative to random 
assignment when evaluating the impacts of charter schools and has been shown to produce valid 
impact estimates that replicate the results of experimental research designs (Tuttle et al. 2013; 
Gill et al. 2015). 7 

Constituting the Kauffman student group. Throughout our analysis, we classify any 
student who was enrolled for at least part of a year in the Kauffman School as a Kauffman 


7 

See Appendix A.4 for more details about the implementation of the propensity -score matching procedure. In 
Appendix B.2, we examine the sensitivity of the results to the use of all students in Kansas City public schools as the 
comparison group rather than those selected by the propensity -score matching procedure. 
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student. Classifying students in this manner defuses the potential criticism that the Kauffman 
School’s effects are overestimated because low-achieving students have left the charter school. 
However, including these students might lead to understating the impact of the Kauffman School 
on student achievement, because students who left the Kauffman School early in the school year 
would have received very little influence from it. Similarly, students from the Kauffman 
School’s first cohort who left after 5th grade are nonetheless kept in the treatment group for the 
6th-, 7th-, and 8th-grade analysis samples. 8 Again, this is a conservative analytic approach that 
eliminates the risk of overestimating the impact, but it means that the full impact on students 
who continue in the Kauffman School for two, three, or four years is likely to be 
underestimated. 9 

Data for our analysis were available for four cohorts of Kauffman students. Cohort I 
students are those who entered the Kauffman School as 5th graders in 201 1-12 (the year the 
Kauffman School opened). Cohort II students are those who entered as 5th graders in 2012-13, 
cohort III students entered as 5th graders in 2013-14, and cohort IV students entered as 5th 
graders in 2014-15. 

In the next chapter, we present impact estimates by the number of years that have passed 
since students first enrolled in the Kauffman School. The four- year impacts are based on the 
outcomes of cohort I students, who were 8th graders in 2014-15. 10 The three-year impacts are 
the average of the Kauffman School’s impact on cohort II students (the 7th graders in 2014-15) 
and its impact on cohort I (the 7th graders in 2013-14). The two-year impacts are the average of 
the Kauffman School’s estimated impact on first three cohorts of students when they were in 6th 
graders. Similarly, the one-year impacts are the average of the Kauffman School’s estimated 
impact on all four cohorts in their 5th-grade year. 

B. Data and descriptive statistics 

In this section, we provide details about the data used in our main analysis of the impact of 
the Kauffman School on student outcomes. We also present a set of descriptive statistics to show 
how Kauffman students compare to students in other schools in the city, in terms of prior 
achievement and demographic characteristics. 

1. Data 

The data we used in our main analyses were provided by the Missouri Department of 
Elementary and Secondary Education. They consisted of MAP test scores in mathematics, ELA, 
and science, along with attendance and suspension data for all students in Missouri who were in 


o 

Only students who entered the Kauffman School in 5th grade are included in the treatment group for this 
analysis. No new students were admitted in 6th or 7th grade during the Kauffman School’s second and third year of 
operation. In year 4, the Kauffman School admitted 22 new students in 6th through 8th grade. These students are 
excluded from the analysis because they are not directly comparable to the other Kauffman students. 

9 In Appendix B.l, we discuss the issue of attrition in more detail and present attrition -adjusted impact 
estimates that approximate the impact of the Kauffman School for 5th-grade students who remain enrolled. 

10 Students who repeated a grade are also included in the calculations. See Appendix B.3 for details. 
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the 5th, 6th, 7th, or 8th grade in the 201 1-12 through 2014-15 school years. 11 We also obtained 
for these students data on their prior (3rd and 4th grade) test scores, prior attendance, prior 
suspensions, and demographic characteristics. We limited our potential comparison group to 
students attending schools in the borders served by the Kansas City Public Schools district 
(KCPS). (See Appendix A.2 for details about the analysis sample selection process.) 

The main results in this report are based on a matched comparison group selected from all 
students attending schools within the borders of KCPS, including other charter schools. This 
group is likely the most relevant for our evaluation, because using these students to construct the 
comparison group provides an impact estimate that can be interpreted in terms of how much 
more or less a Kansas City student would be expected to achieve if that student were to enroll in 
the Kauffman School rather than a typical Kansas City school. However, given the large number 
of charter schools serving students in the Kansas City area, comparing the Kauffman School’s 
impacts on student achievement to those of other Kansas City charter schools might also be of 
interest. Thus, we report the results in three ways: using comparison groups of students from 
(1) all public schools in Kansas City (the primary impact estimates), (2) district-operated (non- 
charter) KCPS schools only, and (3) other charter schools within Kansas City only. 

Data on one or more of the variables used as baseline controls are missing for many 
students. About 12 percent of the students we could potentially include in our analysis sample in 
the most recent year of data are missing data on one or more of the baseline control variables. To 
avoid dropping them from the analysis, we employed a multiple imputation procedure to 
estimate their missing baseline values. 12 We also analyzed the data without using imputation and 
found similar results (Appendix B.4). 

2. Descriptive statistics: What types of students attend the Kauffman School? 

Full descriptive statistics for each cohort of Kauffman students, compared with students in 
other Kansas City public schools, are presented in Appendix A, tables A.2 through A.5. On 
average, 4th-grade MAP scores of Kauffman students were below the statewide average in both 
mathematics and ELA. Students at the Kauffman School were also predominantly from low- 
income and minority families: across the first four cohorts, at least 80 percent of Kauffman 
students were eligible for free or reduced-price lunches in 4th grade, and at least 85 percent were 
black or Hispanic. 

Although, on average, the Kauffman students performed below the state average on the 4th- 
grade MAP, they had higher 4th-grade MAP test scores than other students in Kansas City public 
schools and were less likely to receive any accommodations on the 4th-grade MAP. 13 Compared 
to students in Kansas City public schools, Kauffman students also were more likely to be black, 
were less likely to be Hispanic, and had slightly higher average 4th-grade attendance rates. In 
general, differences tended to be larger relative to Kansas City district schools and smaller 


1 1 In previous years, the name of the ELA assessment was Communication Arts. For simplicity, in this report 
we will use “ELA” to refer to both the 2014-2015 exam and the Communication Arts exam from prior years. 

12 

Appendix A. 3 contains more details about our imputation procedure. 

1 3 

Examples of test accommodations include extended test time, individual testing, and oral reading of test 
questions. 
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relative to Kansas City charter schools. Kauffman students were generally similar to other 
Kansas City public school students with respect to 4th-grade free or reduced-price lunch and 
disability rates, though there were some significant differences across cohorts. 

Because Kauffman students differed from the average student in Kansas City public schools, 
if we included all Kansas City students in the analysis comparison group, there would have been 
significant differences in baseline characteristics between Kauffman and comparison students. 
These differences could lead to concerns about bias in the impact estimates, because students 
who differ with respect to baseline characteristics may be expected to show different rates of 
growth. We therefore used a matching procedure to ensure that the comparison students were 
similar to Kauffman students with respect to baseline characteristics. Appendix A, Table A. 8 
provides descriptive statistics for the matched comparison groups relative to each cohort of 
Kauffman students. By design, there are no significant differences in baseline characteristics 
between the Kauffman and matched comparison groups. Because we were unable to find a match 
for all students, some Kauffman students are not included in the main analyses. For instance, 
when creating a comparison group similar to all KCPS students in 2014-15, we were unable to 
find a match for 9 percent of Kauffman students. Appendix A, Table A.8 shows details on the 
number of Kauffman students excluded for this reason. Caution should therefore be used when 
interpreting the impact results, because it is possible that they are not representative of the 
achievement of all Kauffman students. However, as a robustness check, we performed an 
analysis that did not use matching but relied exclusively on statistical adjustments to account for 
baseline differences between Kauffman and comparison students. This analysis included all 
Kauffman and comparison students with nonmissing data; the results were similar to the main 
findings. See Appendix B.2 for details. 
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III. THE IMPACT OF THE KAUFFMAN SCHOOL ON STUDENT ACHIEVEMENT 


In this chapter we report the impact of the Kauffman School on student achievement as 
measured by the MAP exams in mathematics (one to three years after enrollment), ELA (one to 
four years after enrollment), and science (one year and four years after enrollment). The number 
of years for which we can measure impacts depends on the state’s testing regimen in each subject 
and grade. We describe various ways of interpreting the impact estimates and place their size in 
the context of findings for other evaluations of charter school effectiveness. We also evaluate 
whether the Kauffman School achieved its goal of 75 percent of students enrolled for three 
consecutive years achieving proficient or advanced scores on the MAP exams. 

A. Impacts on state test scores 

The impact estimates for the Kauffman School on student achievement in each MAP subject 
are displayed in Table III.l. The results are based on regression models that include the 
Kauffman students and matched comparison students and control for small remaining differences 
in prior achievement and other baseline characteristics. 14 As noted previously, any student who is 
enrolled in the Kauffman School as a 5th grader for at least part of the school year is included in 
the Kauffman group for all impact estimates. The impact estimates should therefore be 
interpreted as the average effect of enrolling in the Kauffman School, accounting for the 
possibility that students may leave. The results are displayed in effect size units, which can be 
interpreted as how many student test score standard deviations higher or lower Kauffman 
students are performing relative to students in the comparison groups (after controlling for 
baseline achievement). 15 Standard errors are displayed in parentheses below each estimate, and 
asterisks indicate whether the estimate is significantly different from zero. 


14 The covariates include all the variables summarized in Table A.2. We also include 3rd-grade mathematics 
and ELA MAP scores, second- and third-order polynomial terms for 4th-grade MAP scores, and indicator variables 
that equal one if a student has imputed prior test scores or imputed attendance or suspension data. 

15 During the 2014-2015 school year, the statewide standard deviations of 8th-grade MAP scores were 96 in 
mathematics, 88 in ELA, and 32 in science; of 7th-grade MAP scores, 97 in mathematics and 92 in ELA; of 6th- 
grade MAP scores, 92 in mathematics and 90 in ELA; and of 5th-grade MAP scores, 82 in mathematics, 84 in ELA, 
and 32 in science. 


9 



III. STUDENT ACHIEVEMENT MATHEMATICA POLICY RESEARCH 


Table 111.1. Impact of Kauffman School on MAP test scores (citywide 

comparison group) 






Mathematics 

ELA 

Science 

Sample size 

One-year impact estimates (5th grade) 

0.22** 

(0.03) 

0.23** 

(0.03) 

0.46** 

(0.04) 

2,956 

Two-year impact estimates (6th grade) 

0.35** 

(0.04) 

0.18** 

(0.04) 

n.a. 

2,205 

Three-year impact estimates (7th grade) 

0.68** 

(0.06) 

0.52** 

(0.06) 

n.a. 

1,114 

Four-year impact estimates (8th grade) 

0.96“ a 

(0.10) 

0.53** 

(0.08) 

0.66“ 

(0.10) 

748 


Notes: This table displays impact estimates in effect size units. The first row presents the average one-year impact 
estimates for cohort I, II, III, and IV 5th graders. The second row presents the average two-year impact 
estimates for cohort I, II, and III 6th graders. The third row presents the three-year impact estimates of 
cohort I and II 7th graders. The fourth row presents the four-year impact estimates of cohort I (the only 
cohort that has completed four years in the Kauffman School). Standard errors are displayed in 
parentheses below each impact estimate. The sample size represents the total number of Kauffman and 
matched comparison students entering each analysis. 

‘‘Significantly different from zero at the 1 percent level. 

a The four-year mathematics impact is based in part on imputed outcome data. See text for details. 

ELA = English language arts; n.a. = not applicable. 

The first row of Table III. 1 shows the amount of additional growth realized by Kaufman 
students relative to matched comparison students in all other Kansas City public schools one year 
after enrollment. The numbers represent the average effect size estimate for the first four cohorts 
of 5th graders. 16 The one-year impact estimates for the Kauffman School are positive and 
statistically significant in mathematics, ELA, and science. Caution should be used when 
interpreting the science estimate, however, because no prior- year science test score was available 
to use in the propensity-score matching procedure or as a control variable in the regressions. The 
statewide science assessment in Missouri is first administered in 5th grade, so the only baseline 
test score variables available for use in the analysis of 5th-grade science impacts are prior scores 
in mathematics and ELA. 

The remaining rows of Table III. 1 report the estimated effect of the Kauffman School on 
student achievement two, three, and four years after enrollment. With the exception of the four- 
year ELA impact estimate, all the impacts increase with longer duration. Collectively, these 
results demonstrate that the impact of the Kauffman School accumulates for students who are 
enrolled for multiple years. 

Approximately 20 percent of 8th-grade students outside the Kauffman School took the 
Algebra I end-of-course exam in place of the 8th-grade mathematics MAP exam. We imputed 
the missing 8th-grade mathematics test scores for these students using their other 8th-grade test 
scores as well as prior mathematics and ELA test scores and demographic characteristics (see 
Appendix A. 3 for details). The four-year mathematics impact estimate should be interpreted with 


16 The effect sizes were estimated separately for each cohort of students. To calculate the impact estimates in 
Table III. 1 , we averaged these effect sizes together, weighting by the number of Kauffman students in the analysis 
sample for each cohort. 
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caution, because the imputation procedure may not provide an accurate estimate of these 
students’ 8th-grade mathematics scores. This could occur if advanced students who were likely 
to do well in mathematics chose to take Algebra I instead of 8th-grade mathematics, and this 
aptitude for mathematics was not fully captured in the student’s 7th-grade mathematics or other 
test scores. Because of this issue, we focus on the three-year impact mathematics estimates when 
discussing the magnitude of the effect of the Kauffman School on student achievement. To 
simplify the comparisons of the Kauffman School impacts with results from other studies, we 
focus on the three-year ELA impact estimates as well. These impact estimates also have the 
advantage of being based on two cohorts of students rather than one. 

In Table III.2, we report the results with two alternative comparison groups. The first half of 
the table reports the effect size estimates for the Kauffman School compared to district-operated 
(non-charter) public schools in KCPS. The impact estimates in all subjects are larger when this 
comparison group is used. The second half of Table III.2 presents the results when the Kauffman 
School is compared to other charter schools in Kansas City. When compared to this group of 
schools, the effect size estimates for the Kauffman School are generally lower, but the estimates 
remain positive and statistically significant for all durations and subjects. Thus, students at the 
Kauffman School are showing significantly higher growth than students in other Kansas City 
charter schools. 


Table 111.2. Alternate estimates of impact of Kauffman School on MAP test 
scores (district and charter school comparisons) 



Mathematics 

ELA 

Science 

Sample size 

Compared to Kansas City district schools 

One-year impact estimates (5th grade) 

0.24** 

(0.04) 

0.25** 

(0.03) 

0.51** 

(0.04) 

2,200 

Two-year impact estimates (6th grade) 

0.35** 

(0.05) 

0.17** 

(0.04) 

n.a. 

1,566 

Three-year impact estimates (7th grade) 

0.77“ 

(0.07) 

0.62“ 

(0.07) 

n.a 

744 

Four-year impact estimates (8th grade) 

1.05“ 

(0.13) 

0.65** 

(0.09) 

0.84“ 

(0.13) 

457 

Compared to Kansas City charter schools 

One-year impact estimates (5th grade) 

0.20** 

(0.04) 

0.23** 

(0.04) 

0.40** 

(0.05) 

1,366 

Two-year impact estimates (6th grade) 

0.32** 

(0.05) 

0.20** 

(0.05) 

n.a 

929 

Three-year impact estimates (7th grade) 

0.57** 

(0.08) 

0.45** 

(0.08) 

n.a 

506 

Four-year impact estimates (8th grade) 

0.84“ 

(0.14) 

0.41“ 

(0.10) 

0.51“ 

(0.12) 

343 


Notes: This table displays impact estimates in effect size units. Standard errors are displayed in parentheses 
below each impact estimate. The sample size represents the total number of Kauffman and matched 
comparison students entering each analysis. 

“Significantly different from zero at the 1 percent level. 

ELA = English language arts; n.a. = not applicable. 
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B. Interpretation of Kauffman School impact estimates 

To assist with the interpretation of the effect size estimates, we converted the effect sizes 
into three alternative units: (1) years of learning growth, (2) the change in state test score 
percentile rank for the average Kauffman student, and (3) the effect size as a percentage of local 
achievement gaps. 

Results as years of learning growth. We can translate the effect sizes presented in the 
previous section into an approximate measure of the years of additional learning growth 
experienced by Kauffman students based on results presented in Bloom et al. (2008). 17 
Translating the results in this way allows us to evaluate whether the Kauffman School is 
achieving its goal of producing on average at least 1.25 years of learning growth for students 
during each year of instruction. 

Performing this conversion on the impact estimates for the main comparison group yields 
the results displayed in Figure III. 1 . These results show that the Kauffman School is meeting its 
aim of producing on average at least 1.25 years of learning in each year of instruction, since the 
average additional growth per year is 0.25 or greater for all impact estimates in each subject. 
Note that caution is warranted when interpreting the results for Kauffman School effect size in 
terms of years of learning, because the accuracy of these conversions depends on how similar 
achievement growth on the MAP exams is to the vertically scaled assessments analyzed in 
Bloom et al. (2008). 18 It is also worth noting that the results in Figure III. 1 provide a 
conservative estimate of whether the Kauffman School is achieving the 1.25 years of learning 
growth goal, because students who are no longer enrolled in the school are included in the 
Kauffman student sample. To obtain a more complete picture of the Kauffman School’s progress 
towards achieving this goal, we also use results from the NWEA exams and STEP literacy 
assessments that are administered to Kauffman students. See Gentile et al. (2015) for details. 


17 See Gleason et al. (2012), Clark et al. (2013), and Tuttle et al. (2013) for examples of other studies 
performing conversions between effect size estimates and years of learning growth. Using a set of widely 
administered vertically scaled assessments, Bloom et al. (2008) estimated that the typical 5th grader grows 0.56 
standard deviations in mathematics, 0.40 standard deviations in ELA, and 0.40 standard deviations in science. They 
also estimated that the typical 6th grader grows 0.41 standard deviations in mathematics 0.32 standard deviations in 
ELA and 0.27 standard deviations in science, whereas a typical 7th grader grows 0.30 standard deviations in 
mathematics, 0.23 standard deviations in ELA, and 0.28 standard deviations in science. A typical 8th grader grows 
0.32 standard deviations in mathematics, 0.26 standard deviations in ELA, and 0.26 standard deviations in science. 
To convert the one-year impact estimates of the Kauffman School into units of additional years of learning, we 
divided the impact estimates by the typical growth of 5th graders in each subject. We used a similar method to 
convert the two- , three-, and four-year impact estimates into additional years of learning growth. For these results, 
we divided the impact estimates by the average of the typical growth across all grades included in each analysis. 

1 8 

If typical achievement growth on the MAP is less than growth on the assessments analyzed in Bloom et al. 
(2008), then this conversion will underestimate the additional years of learning growth achieved by Kauffman 
students and vice versa. The scale of the MAP assessments is based, in part, on the Terra Nova exams, giving the 
MAP some of the characteristics of a vertically scaled exam. Thus, in principle, we could use average growth on the 
MAP in place of the numbers from Bloom et al. (2008). However, there are known issues with the MAP vertical 
scale when students show no growth on average between grades 5 and 6 (CTB McGraw-Hill 2012). Therefore, we 
did not attempt to use the vertical scale of the MAP to convert effect sizes into units of years of learning. 
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Figure 111.1. Kauffman School estimates of additional years of learning 
growth on MAP exams 



u Mathematics u English language arts u Science 


Note: The additional growth for all impact estimates is significantly different from zero. 

a The four-year mathematics impact is based in part on imputed outcome data. See text for details. 

Results as test score percentile ranks. As a second interpretation approach, we report the 
change in state test score percentile ranks that the average Kauffman student would expect to 
achieve as a result of attending the Kauffman School rather than an average Kansas City school. 
In 4th grade, before entering the Kauffman School, the average Kauffman student from the first 
and second cohorts was at the 39th percentile in both the state mathematics test score distribution 
the state ELA test score distribution. 19 Three years after enrollment at the Kauffman School, the 
average student moved to the 65th percentile in mathematics and the 60th percentile in ELA. 20 
On average, students enrolled at the Kauffman School moved from substantially below the state 
average to above average three years after enrollment. 

Results as a percentage of local achievement gaps. The Kauffman School effect size 
estimates can also be reported as a percentage of the local black- white test score gap (Figure 
HI. 2). These percentages provide a sense of how much of that achievement gap is being closed 
three years after enrollment in the Kauffman School. The three-year impact estimates are 
equivalent to 84 percent of the 7th-grade black-white test score gap in both mathematics and 
ELA for Kansas 


19 These calculations are based on the current analysis sample of cohort I and II students only, because this is 
the sample used to calculate the three-year impact estimates. 

20 

The percentile ranks three years after enrollment at the Kauffman School were calculated by taking the 
average 4th-grade z- scores of Kauffman students and adding the three -year effect size estimates. These calculations 
assume that the percentile rank of the average student in Kansas City does not change over time. 
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City students. 21 The four-year science impact estimate for the Kauffman School is equivalent to 
58 percent of the black-white test score gap. 22 These results indicate that the Kauffman School is 
making significant progress toward reducing achievement gaps for minority students. 

Figure III. 2. Kauffman School impact estimates as a percentage of the local 
black-white test score gap 
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Note: All impact estimates are significantly different from zero. 

C. Comparison of Kauffman School estimated impacts to those of charter 
schools nationwide 

The main three-year effect size results (based on the citywide comparison group) are larger 
than the average impacts estimated for high-performing charter schools in other studies and 
substantially greater than average impact estimates for charter schools nationwide. Here we 
discuss studies of three groups of charter schools widely acknowledged as producing substantial 
achievement growth for students: Boston, KIPP, and New York City charter schools. We also 
report the results of three nationwide studies of charter schools that focused on (1) charter 
schools with admission lotteries, (2) the average urban charter school in 41 regions analyzed by 
the Center for Research on Education Outcomes (CREDO 2015), and (3) charter school 
management organizations (CMOs). The comparisons are summarized in Figure III. 3. 


84% 84% 



21 

The average 7th-grade mathematics z-score for non-Kauffman black students in Kansas City in mathematics 
is -0.772, and the average z-score for white students is 0.039. The corresponding z-scores in ELA are -0.683 for 
black students and -0.065 for white students. These z-scores are based on test score data from the spring of 2015 to 
provide an estimate of the current black- white test score gap. 

22 

The average 8th-grade science z-score for non-Kauffman black students in Kansas City is -1.049; the 
corresponding average z-score for white students is 0.080. These z-scores are based on data from spring 2015. 
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Figure III. 3. Charter school three-year impact estimates from various studies 



School charters City charters charters income urban 

charter charter 
lotteries school study 

h Mathematics h Reading/English language arts 

Note: Figure III. 3 contains three-year effect size estimates for the average Boston charter school reported in 

Abdulkadiroglu et al. (2009), the average KIPP charter school analyzed by Tuttle et al. (2013), the average 
New York City charter school in grades 4 through 8 reported in Hoxby, et al. (2009), the average charter 
school with a lottery admission process serving a large fraction of low-income students analyzed by 
Gleason et al. (2010), the average urban charter school in the 41 regions analyzed by the Center for 
Research on Education Outcomes (CREDO 2015), and the average school in the CMOs studied by 
Furgeson et al. (2012). 

CMO = charter school management organization; CREDO = Center for Research on Education Outcomes; KIPP = 
Knowledge Is Power Program. 

The performance of the Kauffman School in mathematics and reading is higher than the 
average estimated impacts of other successful charter school programs. A study of Boston 
charter schools (Abdulkadiroglu et al. 2009) showed an estimated average three-year effect size 
for these schools of 0.53 in mathematics and 0.28 in reading. 23 The results presented by Hoxby et 
al. (2009) imply average three-year effect sizes of 0.36 in mathematics and 0.27 in reading for 
New York City charter schools in grades 4 through 8. 24 A study of KIPP charter middle 
schools — widely recognized as high-performing — reported average three-year impact estimates 
of 0.36 in mathematics and 0.21 in reading (Tuttle et al. 2013). The Kauffman School is 
estimated, on average, to have larger impacts on student achievement than these high-performing 
charter school programs. It is important to note that Figure III. 3 displays the average impacts of 
the charter schools in these groups. Some individual schools or subsets of these groups have 
larger estimated impacts than the Kauffman School. For example, the three-year impact 
estimates for over-subscribed Boston charter middle schools analyzed in Abdulkadiroglu et al. 


23 

The three-year impact estimates reported in this section are generally obtained by tripling the average annual 
impact estimates reported by the authors. The exceptions to this are the KIPP study, the charter lottery study, and the 
CMO study. In the KIPP and CMO studies, the authors reported three-year impact estimates separately from one- 
year estimates. The three-year impact estimates for the charter lottery study were obtained by increasing the two- 
year estimates by 50 percent. 

24 

The estimated impacts of New York City charter schools on lower elementary school grades are smaller 
than in grades 4 through 8. When lower elementary school grades are included in the calculations, the average three - 
year impact estimates for New York City charter schools are 0.27 in mathematics and 0.18 in ELA. 
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(2009) are equivalent to 0.51 standard deviations in reading and 1.6 standard deviations in 
mathematics. 

The achievement growth of students at the Kauffman School is substantially higher than the 
average growth of students at broader samples of charter schools nationwide (Figure III.3). 
Gleason et al. (2010) analyzed a sample of oversubscribed charter middle schools with lottery 
admission processes. The results indicate an average three-year impact estimate of 0.27 in 
mathematics and zero in reading for charter schools serving a large fraction of low-income 
students. 25 Across urban charter schools in the 41 regions included in the Center for Research on 
Education Outcomes (CREDO) analysis, the average impact was 0.17 in mathematics and 0.12 
in reading (CREDO 2015). The average charter school in the CMOs analyzed by Furgeson et al. 
(2012) was estimated to produce three-year effect sizes of 0.15 in mathematics and 0.05 in 
reading. 

The three-year impact estimates for the Kauffman School are approximately double the size 
of the two-year estimates. When the Kauffman School two-year impact estimates are compared 
to those reported in other studies of charter schools, the estimates fall within the range of other 
highly successful charter programs but are still substantially larger than the impact estimates 
from national charter school studies (Figure III.4). 

Figure 111.4. Charter school two-year impact estimates from various studies 



School charters City charters charters income urban 

charter charter 
lotteries school study 

a Mathematics h Reading/English language arts 

Notes: Figure III. 4 contains two-year effect size estimates for the average Boston charter school reported in 

Abdulkadiroglu et al. (2009), the average KIPP charter school analyzed by Tuttle et al. (2013), the average New 
York City charter school in grades 4 through 8 reported in Hoxby et al. (2009), the average charter school with a 
lottery admission process serving a large fraction of low-income students analyzed by Gleason et al. (2010), the 
average urban charter school in the 41 regions analyzed by the Center for Research on Education Outcomes 
(CREDO 2015), and the average school in the CMOs studied by Furgeson et al. (2012). 

CMO = charter school management organization; CREDO = Center for Research on Education Outcomes; KIPP = 

Knowledge Is Power Program. 


25 

Gleason et al. (2010) report negative but statistically insignificant impact estimates based on the full set of 
charter schools in their sample (not only those serving low-income students). 
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Because there are fewer available data for science achievement than for mathematics and 
reading, relatively few studies of charter school effectiveness report impact estimates in 
science. 26 The Kauffman School’s estimated four-year science impact compares favorably to the 
science impacts of KIPP middle schools, which are estimated to have a cumulative average 
impact of 0.33 standard deviations in science for students three to four years after enrollment 
(Tuttle et al. 2013). 

D. Goal that 75 percent of students score proficient or advanced on MAP 
exams 

One goal of the Kauffman School is that at least 75 percent of students enrolled for three 
consecutive years score in the proficient or advanced performance range on “all” subjects of the 
MAP test. According to school staff, the original intent of the goal is that 75 percent of students 
would achieve proficiency on each state test administered to its students. The proficiency rates of 
the first two cohorts of Kauffman students enrolled for three consecutive years are summarized 
in Table III. 3. The first column displays the percentage of these students who scored proficient 
or advanced on the MAP exams taken in the spring before they entered the Kauffman School 
(i.e., in 4th grade). This column provides an indication of how ambitious the 75 percent goal is, 
as only 39 percent of incoming cohort I and II had achieved proficient or advanced in 
mathematics and 36 percent in ELA on their prior-year MAP exams. 

The Kauffman School did not meet the goal of 75 percent of students achieving at the 
proficient or advanced levels after three years of consecutive enrollment. Among students who 
were enrolled at the Kauffman School for three consecutive years, 68 percent achieved proficient 
or advanced on the mathematics, and 67 percent on the ELA MAP exams. To provide additional 
detail about the progress Kauffman students made toward this goal, we report the results 
separately for cohort I and cohort II students. 


Table 111.3. Percentage of Kauffman students scoring proficient or advanced 
on MAP exams after three years of continuous enrollment 



Proficient/advanced 
at time of entry 

Proficient/advanced after three 
years of enrollment 

Cohort 1 and II students combined 

Mathematics MAP (%) 

39 

68 

ELA MAP (%) 

36 

67 

Both mathematics and ELA MAP (%) 

26 

57 

Sample size 

140 

152 

Cohort 1 students 

Mathematics MAP (%) 

32 

66 

ELA MAP (%) 

30 

55 

Both mathematics and ELA MAP (%) 

20 

51 


96 

Hoxby et al. (2009) reports average annual impacts of 0.23 standard deviations in science for New York 
City charter schools in grades 5 through 8. However, that impact is estimated with a large standard error and is not 
statistically significant (p-value = 0.14). Multiplying this annual impact estimate by four would therefore be unlikely 
to provide a meaningful comparison with the four-year science impact of the Kauffman School. 
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Sample size 

74 

74 

Cohort II students 

Mathematics MAP (%) 

45 

68 

ELA MAP (%) 

44 

78 

Both mathematics and ELA MAP (%) 

32 

62 


Sample size 66 78 


Notes: The sample includes 152 cohort I and cohort II students who were enrolled at the Kauffman School for 

three consecutive years. The scores at time of entry are based on 4th-grade MAP scores for 134 students 
and 3rd-grade MAP scores for 6 students who skipped a grade when entering the Kauffman School. 
Twelve cohort II students are missing baseline MAP exam scores. The scores after three years of 
enrollment are based on 7th-grade MAP exams for 146 students and 6th-grade MAP exams for 6 students 
who repeated a grade while at the Kauffman School. 

ELA = English language arts 


Table III.4 displays the percentage of Kauffman students scoring proficient or advanced 
after four years of continuous enrollment. The Kauffman School achieved the goal of 75 percent 
proficiency on each state test for these students, with 81 percent of scoring proficient or better in 
mathematics, 76 percent in ELA, and 78 percent in science. Overall, 64 percent of students 
reached proficiency on all three exams after four years of enrollment. 


Table 1 1 1.4. Percentage of Kauffman students scoring proficient or advanced 
on MAP exams after four years of continuous enrollment 



Proficient/advanced 
at time of entry 

Proficient/advanced after four 
years of enrollment 

Mathematics MAP (%) 

32 

81 

ELA MAP (%) 

27 

76 

Science MAP (%) 

n.a. 

78 

All available MAP assessments (%) 

19 

64 

Sample size 

59 

59 


Notes: The sample includes 59 cohort I students who were enrolled at the Kauffman School for four consecutive 

years. The scores at time of entry are based on 4th-grade MAP scores taken in spring 201 1 for 55 students 
and 3rd-grade MAP scores taken in spring 201 1 for 4 students who skipped a grade when entering the 
Kauffman School. The scores after four years of enrollment are based on 8th-grade MAP exams taken in 
spring 2015 for 54 students, and 7th-grade MAP exams taken in spring 2015 for 5 students who repeated a 
grade during their time at the Kauffman School. These 5 students are included in the calculation of overall 
proficiency rates in the first row of the table, with results based only on mathematics and ELA scores, 
because 8th grade science MAP scores are unavailable for them. 

ELA = English language arts; n.a. = not applicable. 
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IV. CHANGE IN EFFECTIVENESS OF THE KAUFFMAN SCHOOL OVER TIME 


The Kauffman School has completed four full years of operation: 2011-12, 2012-13, 2013- 
14, and 2014-15. We can use data on 5th, 6th, and 7th graders to test whether the estimated 
impacts of the Kauffman School have changed over time. We might expect to see an increase in 
the estimated impact over time, because other studies have found that charter schools often show 
increases in performance after the first year (Gill et al. 2007; Sass 2006). 

Table IV. 1 compares the estimated one-year (5th grade) Kauffman School impacts for 
cohorts I, II, III, and IV. Tables IV.2 and IV. 3 display the corresponding comparisons of impacts 
for 6th and 7th graders across years. 

Looking across the three tables, a clear pattern emerges: although there is no evidence of 
any statistically significant changes in impacts over the first three years of the school’s operation, 
impacts for the fourth year (2014-15) exceed those of preceding years in most of the 
comparisons. One-year impacts for 5th graders are higher in 2014-15 than in the preceding year 
in reading and mathematics (but not statistically distinguishable in science). Two-year impacts 
for 6th graders are higher in mathematics for the most recent cohort relative to the preceding one 
(but not distinguishable in reading). Three-year impacts for 7th graders are higher in both 
English and mathematics for the most recent cohort relative to the preceding cohort. In short, 
even though the Kauffman School was producing significant achievement impacts in earlier 
years, it appears to have substantially accelerated its achievement impacts in 2014-15. 

The significant increase in estimated impacts across grades and subjects in the 2014-15 
school year may be related to the change in standards tested on the MAP exams to align with the 
Common Core State Standards. The Kauffman School had already changed its curricula to align 
with the Common Core standards prior to 2014-15, which likely positioned students to perform 
well on the new MAP exams. It is also possible that the larger impacts in year 4 were related to 
other factors, such as increases in teacher or school effectiveness. 
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Table IV.1. Comparison of one-year MAP test score impacts: Cohort I, II, III, 
and IV 5th graders 



Cohort 1 

5th graders 
(2011-12) 

Cohort II 

5th graders 
(2012-13) 

Cohort III 

5th graders 
(2013-14) 

Cohort IV 

5th graders 
(2014-15) 

5th-grade mathematics 

0.13 

0.15 

0.12 

0.42** 

effect size 

(0.06) 

(0.07) 

(0.04) 

(0.06) 

5th-grade ELA effect size 

0.06 

0.18 

0.13 

0.44** 


(0.07) 

(0.06) 

(0.04) 

(0.06) 

5th-grade science effect 

0.40 

0.54 

0.43 

0.52 

size 

(0.08) 

(0.07) 

(0.04) 

(0.07) 

Sample size 

677 

617 

948 

714 


Notes: This table displays impact estimates in effect size units. Standard errors are displayed in parentheses 
below each impact estimate. The sample size represents the total number of Kauffman students and 
matched comparison students entering each analysis. 


**Significantly different from the prior cohort at the 1 percent level. 
ELA = English language arts. 


Table IV.2. Comparison of two-year MAP test score impacts: Cohort I, II, and 
III 6th graders 



Cohort 1 

Cohort II 

Cohort III 


6th graders 
(2011-2013) 

6th graders 
(2012-2014) 

6th graders 
(2013-2015) 

6th-grade mathematics effect 

0.33 

0.20 

0.43* 

size 

(0.08) 

(0.07) 

(0.06) 

6th-grade ELA effect size 

0.18 

0.20 

0.17 


(0.07) 

(0.07) 

(0.06) 

Sample size 

596 

585 

1,024 


Notes: This table displays impact estimates in effect size units. Standard errors are displayed in parentheses 
below each impact estimate. The sample size represents the total number of Kauffman students and 
matched comparison students entering each analysis. 

*Significantly different from the prior cohort at the 5 percent level. 

ELA = English language arts. 
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Table IV.3. Comparison of three-year MAP test score impacts: Cohort I and II 
7th graders 



Cohort 1 7th graders 
(2011-2014) 

Cohort II 7th graders (2012- 
2015) 

7th-grade mathematics effect size 

0.57 

0.80* 


(0.07) 

(0.09) 

7th-grade ELA effect size 

0.41 

0.66* 


(0.08) 

(0.09) 

Sample size 

534 

580 


Notes: This table displays impact estimates in effect size units. Standard errors are displayed in parentheses 
below each impact estimate. The sample size represents the total number of Kauffman students and 
matched comparison students entering each analysis. 


*Significantly different from the prior cohort at the 5 percent level. 
ELA = English language arts. 
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V. THE IMPACT OF THE KAUFFMAN SCHOOL ON STUDENT ATTENDANCE 
AND SUSPENSIONS 


In this chapter, we present the impacts of the Kauffman School on student attendance and 
suspensions during the 2014-15 school year. Keep in mind that the analysis of suspensions 
cannot distinguish effects driven by differences in student behavior from effects driven by 
differences in the enforcement of school policies or reporting practices. If Kauffman students are 
more likely than students in other schools to be suspended, it could be due to an increased 
frequency of infractions or because the Kauffman School issues suspensions for different kinds 
of behavior than do other Kansas City schools. 

The set of comparison students used to analyze attendance and suspension outcomes is the 
same as the set used to analyze MAP achievement in Chapter III. We also used the same set of 
baseline control variables, including 4th-grade attendance and suspension information. We 
analyzed the attendance and suspension outcomes separately by cohort and by grade to highlight 
differences that arise over time and across grades. We used the attendance rate as the outcome in 
the attendance models and present three sets of results for suspension outcomes. The state of 
Missouri collects suspension data separately for in-school suspensions and out-of-school 
suspensions. For the initial suspension analysis, we combined these data into one variable, 
indicating whether a student received either type of suspension. Our aim in combining these data 
was to create a variable that would be as comparable as possible across schools, because 
different schools have different standards for the kinds of disciplinary infractions that warrant in- 
school and out-of-school suspensions. To provide additional information on the source of the 
Kauffman School suspension impacts, we also present separate results where indicators for ever 
receiving an in-school or out-of-school suspension are used as outcome variables. (Appendix A.l 
provides further details on the construction of the attendance and suspension variables.) 

The distribution of suspensions is skewed, with the majority of students receiving no 
suspensions. To simplify the analysis for the main suspension impact results, our outcome of 
interest is whether a student ever received a suspension during the year. 27 Before presenting the 
impact estimates of the Kauffman School, we present descriptive statistics about attendance and 


27 

Because the suspension outcome is a binary rather than a continuous variable, we use a logit model in place 
of the linear regression to implement the analysis. 
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suspensions at the Kauffman School and other schools in the Kansas City school district during 
the 2014-15 school year. That descriptive information is displayed in Table V.l. 

Table V.l. Attendance and suspension rates for Kauffman and all Kansas City 


students during 2014-15 



Kauffman students 

Kansas City students 

5th graders 

Attendance rate (%) 

96.2 (3.5) 

94.5 (4.9)** 

Received one or more suspensions (%) 

31.0 (46.4) 

18.4 (38.7)** 

One or more in-school suspensions (%) 

20.3 (40.3) 

7.3 (26.1)** 

One or more out-of-school suspensions (%) 

23.5 (42.5) 

14.3 (35.0)** 

Sample size 

187 

1,585 

6th graders 

Attendance rate (%) 

96.0 (3.5) 

94.6 (4.9)** 

Received one or more suspensions (%) 

29.6 (45.8) 

24.6 (43.1) 

One or more in-school suspensions (%) 

17.3 (37.9) 

11.9 (32.4) 

One or more out-of-school suspensions (%) 

19.6 (39.8) 

18.2 (38.6) 

Sample size 

179 

1,592 

7th graders 

Attendance rate (%) 

95.4 (4.7) 

93.6 (5.9)** 

Received one or more suspensions (%) 

33.7 (47.6) 

30.3 (46.0) 

One or more in-school suspensions (%) 

22.9 (42.3) 

15.1 (35.9) 

One or more out-of-school suspensions (%) 

21.7 (41.5) 

22.5 (41.8) 

Sample size 

83 

1,472 

8th graders 

Attendance rate (%) 

94.6 (5.2) 

93.2 (6.2)* 

Received one or more suspensions (%) 

48.9 (50.3) 

34.6(47.6)* 

One or more in-school suspensions (%) 

35.2 (48.0) 

21.6 (41.2)* 

One or more out-of-school suspensions (%) 

29.5 (45.9) 

24.2 (42.8) 

Sample size 

88 

1,455 

Average across grades 

Attendance rate (%) 

95.7 (4.1) 

94.2 (5.3)** 

Received one or more suspensions (%) 

33.9 (47.0) 

25.0 (42.9)** 

One or more in-school suspensions (%) 

22.2 (41.3) 

12.4 (32.6)** 

One or more out-of-school suspensions (%) 

22.9 (42.1) 

18.5 (38.7)* 

Sample size 

537 

6,104 


Notes: Standard deviations are presented next to means in parentheses. The bottom section of the table displays 

the average across grade levels, weighted by the number of Kauffman students in each grade. 

‘Significantly different from Kauffman students at the 5 percent level. 

“Significantly different from Kauffman students at the 1 percent level. 

The average attendance rate of Kauffman students across all grades was significantly higher 
than that of other Kansas City students. The Kauffman School also suspended students at a 
significantly higher rate during 2014-15, compared with other schools in Kansas City. The 
descriptive statistics presented in Table V.l are based on the full sample of Kauffman and 
Kansas City students in 5th through 8th grades during 2014-15. In Table V.2, we present the 
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estimated impacts of the Kauffman School on attendance and suspensions, based on the sample 
of matched comparison students. In Appendix B, Table B.8, we report the attendance and overall 
suspension impact estimates when the two alternative comparison groups (charter-school and 
KCPS students) are used. 

Table V.2. Impact of Kauffman School on attendance and suspensions during 
2014-15 



5th graders 

6th graders 

7th graders 

8th graders 

2014-15 

average 

Attendance rate (%) 

1.13** 

(0.40) 

0.91** 

(0.35) 

0.68 

(0.71) 

0.42 

(0.59) 

0.87** 

(0.23) 

Probability of being suspended (%) 

13.1** 

(4.4) 

3.82 

(3.9) 

4.0 

(6.5) 

14.0** 

(5.3) 

8.85** 

(2.4) 

Probability of in-school suspension 

14.2** 

5.69 

7.82 

14.2** 

10.5** 

(%) 

(3.7) 

(3.3) 

(5.6) 

(4.7) 

(2.1) 

Probability of out-of-school 

9.39* 

0.84 

1.12 

4.48 

4.51 

suspension (%) 

(4.3) 

(3.6) 

(6.2) 

(5.3) 

(2.3) 

Sample size 

714 

1,024 

580 

748 

3,066 


Notes: The suspension results are marginal effects from logit models in which the outcome variable is an indicator 
for receiving a suspension during the year. Standard errors are displayed in parentheses beside each 
impact estimate. The sample size represents the total number of Kauffman students and matched 
comparison students entering each analysis. The fifth column represents a weighted average (by the 
number of Kauffman students) of the impacts across grade levels. 

‘Significantly different than zero at the 5 percent level. 

“Significantly different than zero at the 1 percent level. 


Impact on attendance. The results show that during the 2014-15 school year, the 
Kauffman School overall had a positive and statistically significant impact on the attendance rate 
of its students. The magnitude of the coefficient indicates that, on average, Kauffman students 
had an attendance rate 0.90 percentage points higher than that of other similar students in Kansas 
City. The estimated impact is positive in all grades, but is statistically significant only for 5th and 
6th graders (as well as for all grades combined). 

Impact on suspensions. The estimated suspension impacts for the Kauffman School are 
also positive, which indicates that Kauffman students are significantly more likely to be 
suspended than similar students in Kansas City. Overall, Kauffman students were approximately 
9 percentage points more likely to receive at least one suspension during the school year. This 
significant overall suspension impact estimate appears to be driven by higher in-school 
suspension rates at the Kauffman School. The school did not have a statistically significant effect 
on the probability that a student would receive an out-of-school suspension. 

To help put the high suspension rates at the Kauffman School into perspective, we present in 
Table V.3 the average number of days suspended among students who receive at least one 
suspension. The average number of days suspended among students ever suspended is not 
significantly different between Kauffman and matched comparison students in any of the three 
grades examined. The average number of days suspended among students receiving at least one 
suspension is between two and three for most grades for both Kauffman and comparison 
students. The results are similar when median days suspended is examined in place of average 
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days: the figure for median days ranges from one to two across grades for both groups. Overall, 
although more students receive suspensions at the Kauffman School, the average number of days 
missed due to suspensions among suspended students is not significantly higher. 


Table V.3. Average number of days suspended for students receiving 
suspensions during 2014-15 



Kauffman students 

Kansas City students 

5th graders 

Mean days suspended 

2.26 (1.85) 

2.06 (1.63) 

Mean days suspended (in-school) 

1.59 (1.13) 

1.54 (1.01) 

Mean days suspended (out-of-school) 

1.64 (0.93) 

1.88 (1.43) 

Sample size 

55 

106 

6th graders 

Mean days suspended 

2.20 (1.73) 

2.33 (2.12) 

Mean days suspended (in-school) 

1.64 (0.95) 

1.96 (2.13) 

Mean days suspended (out-of-school) 

1.82 (1.27) 

1.89 (1.25) 

Sample size 

47 

213 

7th graders 

Mean days suspended 

2.58 (2.06) 

2.66 (2.70) 

Mean days suspended (in-school) 

2.23 (1.91) 

2.43 (2.49) 

Mean days suspended (out-of-school) 

1.64 (0.93) 

1.96 (1.61) 

Sample size 

26 

161 

8th graders 

Mean days suspended 

2.54 (2.36) 

3.20 (3.03) 

Mean days suspended (in-school) 

1.95 (1.82) 

2.91 (2.78)* 

Mean days suspended (out-of-school) 

1.86 (1.67) 

2.02 (1.45) 

Sample size 

42 

232 

Average across grades 

Mean days suspended 

2.36 (1.99) 

2.51 (2.34) 

Mean days suspended (in-school) 

1.80 (1.45) 

2.15 (2.12) 

Mean days suspended (out-of-school) 

1.74 (1.22) 

1.93 (1.42) 

Sample size 

170 

712 


Notes: Standard deviations are presented next to means in parentheses. The sample size represents the total 

number of Kauffman students and matched comparison students with at least one suspension (in-school or 
out-of-school). The bottom section of the table displays the average across grade levels for days missed 
due to overall, in-school, and out-of-school suspensions, weighted by the number of Kauffman students 
with at least one suspension in that category. 

‘Significantly different from Kauffman students at the 5 percent level. 


The fact that the average number of days missed among Kauffman students who receive 
suspensions is less than three helps shed light on how the Kauffman School may be having large 
positive impacts on student achievement while suspending students at high rates. The additional 
instructional time at the Kauffman School resulting from the extended school day and year is 
much longer than three days. Kauffman students receive the equivalent of approximately five 
weeks of additional schooling per year relative to other public school students in Kansas City. 


26 



V. THE IMPACT OF THE KAUFFMAN SCHOOL ON STUDENT ATTENDANCE 
AND SUSPENSIONS 


MATHEMATICA POLICY RESEARCH 


Since the average number of days missed due to suspensions is small relative to the additional 
instructional time at the Kauffman School (less than 3 days missed compared to approximately 5 
weeks of additional schooling), it is possible that the high suspension rates are not substantially 
detracting from learning. 

Finally, caution is warranted when interpreting the suspension impact estimates for the 
Kauffman School. The positive and significant impacts could be due to (1) stricter discipline 
policies at the Kauffman School, which might result in the issuing of suspensions for less-severe 
infractions than at other schools; (2) the longer school day and school year at the Kauffman 
School, which provides more opportunities for suspensions to be issued; or (3) an increased 
frequency of infractions by Kauffman students than comparison students. 

Changes in attendance and suspension impacts across years. 

Table V.4 compares the schoolwide attendance and suspension impacts for each year of the 
Kauffman School’s operation. The impact on attendance rates was not significantly different in 
2014-15 compared to 2013-14. The impacts on suspension rates were significantly lower in 
2014-15 for overall, in-school, and out-of-school suspensions compared to 2013-14. 


Table V.4. Comparison of impacts of the Kauffman School on attendance and 
suspensions across years 



2011-12 average 

2012-13 average 

201 3-1 4 average 

2014-15 average 

Attendance rate (%) 

-0.83 (0.48) 

0.87** (0.31) 

0.72 (0.27) 

0.87 (0.23) 

Probability of being 
suspended (%) 

13.4 (5.2) 

7.22 (3.5) 

24.7** (2.6) 

8.85** (2.4) 

Probability of in-school 
suspension (%) 

0.27 (3.9) 

-1.27 (2.2) 

24.8** (2.6) 

10.5** (2.1) 

Probability of out-of-school 
suspension (%) 

14.2 (4.7) 

8.97 (3.4) 

16.6 (2.8) 

4.51** (2.3) 

Sample size 

677 

1,213 

2,067 

3,066 


Notes: The suspension results are marginal effects from logit models in which the outcome variable is an indicator 

for receiving a suspension during the year. Standard errors are displayed in parentheses beside each 
impact estimate. The sample size represents the total number of Kauffman students and matched 
comparison students entering each analysis. 

**Significantly different from the prior school year’s outcomes at the 1 percent level. 
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A. Key findings 

The Kauffman School has ambitious goals for its students: accelerated learning and high 
levels of attendance. The results from our analysis of MAP assessments and attendance data 
show that the Kauffman School is achieving these goals. Though the Kauffman School is 
suspending its students at a higher rate than similar students at other schools in Kansas City, the 
suspension rate in 2014-15 was lower than in the previous year, and the out-of-school 
suspension rate was not distinguishable from the rate for the comparison students. In addition, 
the days missed due to suspensions is small relative to the additional instructional time received 
by Kauffman students. 

Achieving academic goals. An analysis of data from the Kauffman School’s first four years 
shows that the Kauffman School achieved its goal of having students grow on average at least 
1.25 years for every year of attendance, in all three subjects tested by the MAP (mathematics, 
ELA, and science). Upon entering, the average student was substantially below the state average 
in terms of mathematics and ELA scores, but within three years performed above the state 
average in both subjects. 

Comparisons to other charter schools. The Kauffman School’s three-year impacts on test 
scores are larger than the average effects of groups of other charter schools known for their 
strong positive impacts on student achievement, such as Boston, KIPP, and New York City 
charter schools. The Kauffman School’s impacts are similar to those of the highest-performing 
KIPP middle schools. 

Achieving attendance goals. The Kauffman School achieved its goal of an average daily 
attendance rate of 95 percent each year during Years 1, 2, 3, and 4 (Gentile et al. 2015). During 
the 2014-15 school year, the Kauffman School had a statistically significant positive impact on 
its students’ rate of attendance, boosting it by 0.9 percentage points. 

Suspension of students. During 2014-15, Kauffman students were 9 percentage points 
more likely to receive at least one suspension compared with other similar students in Kansas 
City. This difference appears to have been driven primarily by a higher rate of in-school 
suspensions; the Kauffman School did not have a significant impact on the rate of out-of- school 
suspensions. Though the Kauffman School suspended students at a significantly higher rate than 
other schools in Kansas City, the average number of days missed due to suspensions was small 


29 


VI. CONCLUSIONS 


MATHEMATICA POLICY RESEARCH 


relative to the additional instructional time received by Kauffman students. The suspension rate 
at the Kauffman School was also lower in 2014-15 than during the previous year. 

B. Next steps for the Kauffman School 

Growing and moving. Year 5 is a year of change and expansion for the Kauffman School, 
which will open the high school for its first group of 9th-grade students. The student body will 
also continue to grow, with the addition of over 200 new 5th graders, along with backfilling at 
the other grade levels. 

Use of the Common Core State Standards. The State of Missouri discontinued use of the 
Common Core-aligned Smarter Balanced exam after 2015 and is in the process of developing 
new content standards. The Kauffman School plans on continuing to use its Common Core- 
aligned ELA and mathematics curricula in year 5, while Missouri is working on developing the 
new standards. 

Maintaining School culture. As the Kauffman School grows, administrators and teachers 
are attending to the issue of how to maintain and expand the Kauffman School’s culture. On the 
new campus, students and staff occupy different buildings based on grade level. This grouping 
has logistical benefits but poses challenges for instilling a common culture across buildings and 
grade levels. Administrators and teachers are working on ways to maintain the Kauffman 
School’s hallmarks: ambitious academic achievement, high attendance rates, and exemplary 
behavior. 
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1. Data preparation details 

The Missouri Department of Elementary and Secondary Education provided data on state 
assessment results and student characteristics for all students enrolled in Missouri public schools 
during our analysis period. Mathematica Policy Research requested student data for all 3rd 
graders in 2010-2014, all 4th graders in 2011-2014, all 5th graders in 2012-2015, all 6th graders 
in 2013-2015, all 7th graders in 2014-15, and all 8th graders in 2015. The state assessment data 
contained Missouri Assessment Program (MAP) scaled scores, proficiency levels, and test 
accommodation information for each student test by year, grade level, and content area. The 
student characteristics data contained demographic, free or reduced-price lunch status, limited 
English proficiency, disability, attendance, and disciplinary information for each student by year 
and school of enrollment, as well as school-level characteristics such as charter school 
classification and school location. 

To link the state assessment and student characteristics data, we reduced both to the unique 
student level. From the assessment data, we removed records in which students had more than 
one unique subject-specific MAP scaled score reported in a given year. From the characteristics 
data, we first removed all records with zero or missing reported attendance and then summed 
attendance and disciplinary variables across each student’s school-specific records to calculate 
student-year totals. We then reduced the data to the student level such that all year-specific 
attendance/disciplinary information was preserved in separate variables, and demographic, free 
or reduced-price lunch status, limited English proficiency, and disability information was taken 
from the student’s 4th-grade record if available, 3rd-grade record if the 4th-grade record was 
unavailable, and 5th-grade record if both 4th- and 3rd-grade records were unavailable. Students 
not found in both the assessment and the characteristics data were dropped from the analysis. 

We created several new variables to facilitate the analyses. We transformed student MAP 
scaled scores into z-scores based on statewide year-, grade-, and subject-specific means and 
standard deviations. We also used enrollment and absence information to create an attendance 
rate measure that we bottom-coded at the year-specific first percentile to remove extreme 
outliers. We used disciplinary information to create yearly indicators of whether students 
received a suspension that year. 1 We then collapsed subject-specific 3rd- and 4th-grade MAP z- 
scores into grade-specific variables by taking each student’s most recent score (by year) within 
grade level for those students repeating grades. We created a single binary test accommodation 
indicator to represent having test accommodations on any 3rd- or 4th-grade MAP test. 

2. Sample Selection 

The Kauffman School group is composed of students who were enrolled in the Kauffman 
School in 5th grade in 2011-12, 2012-13, 2013-14, or 2014-15 for at least part of the school 
year. 2 The Kansas City district schools comparison group is composed of students enrolled in the 


1 All analyses use 4th-grade attendance and suspensions as control variables. If 4th-grade information on these 
variables was missing, then 3rd-grade values were used instead. 

2 

For the first time in 2014-2015, the Kauffman School backfilled enrollment by accepting new students in 
6th, 7th, and 8th grade who were not previously enrolled in the school in 5th grade. These students were excluded 
from our analysis because the amount of time they spent at the Kauffman School is not comparable to other students 
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Kansas City 33 School District in 5th grade in our analysis years during at least part of the school 
year who were not included in the Kauffman School group. The Kansas City charter schools 
comparison group includes only those students who were enrolled in 5th grade for all or part of 
the school year in a Kansas City charter school. The all Kansas City public schools comparison 
group contained all students in either of the other two comparison groups. We developed a list of 
charter schools using information on school location and background research on charter schools 
identified as being located in Kansas City (Missouri Department of Elementary and Secondary 
Education 2015) and enrolling 5th-grade students in our analysis years. 

In addition to these restrictions, we excluded any Kauffman students missing any outcome 
MAP test scores or all 3rd- and 4th-grade MAP test scores. 3 We also dropped any comparison 
students missing all 3rd- and 4th-grade MAP test scores or any outcome scores from the final 
analysis sample. As a result of these restrictions, for analyses based on data from 2014-15, we 
dropped 14 Kauffman students (14 percent) from the cohort I analysis sample, 19 (19 percent) 
from the cohort II analysis sample, 24 (12 percent) from the cohort III analysis sample, and 23 
(11 percent) from the cohort IV analysis sample. With respect to the full comparison group, we 
dropped 24 percent of otherwise eligible students from the cohort I main analysis sample, 20 
percent from the cohort II analysis sample, 15 percent from the cohort III analysis sample, and 
12 percent from the cohort IV analysis sample. Johnson et al. (2016) provides details about the 
percentage of students dropped from the analyses based on data from 2013-14. Finally, we 
excluded from the comparison group any students who were enrolled for part of any school year 
at the Kauffman School (and included them in the Kauffman School group). Table A.l provides 
the numbers of students included in the Kauffman School and comparison groups for each grade 
and cohort in our analysis. 


Table A.l. Number of students in each comparison group in 2014-15 


Final study group 

Cohort 1 8th 
graders 

Cohort II 

7th graders 

Cohort III 
6th graders 

Cohort IV 
5th graders 

Kauffman students 

88 

83 

179 

187 

All Kansas City public schools comparison group 

1,455 

1,472 

1,592 

1,585 

Kansas City district schools comparison group 

832 

820 

965 

915 

Kansas City charter schools comparison group 

629 

665 

645 

691 


We display the baseline average characteristics of all students included in the Kauffman 
School and comparison groups for cohort I students in Table A.2, cohort II students in Table A.3, 
cohort III students in Table A.4, and cohort IV students in Table A.5. These tables show that 
Kauffman students tend to differ significantly from students enrolled in Kansas City public 
schools on several key baseline measures. Kauffman students had significantly higher average 
4th-grade MAP test scores than students in other Kansas City schools, though all groups had test 


in the same grade. Some 5th-grade students were also enrolled in the school mid-year during 2014-2015. These 
students are included in the analysis because they spent part of their 5th-grade year enrolled at the Kauffman School. 

Students who transfer to different school districts in Missouri will generally remain in our sample, but 
students who leave the state will be excluded due to missing outcome test scores. 
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scores below the statewide average. 4 Kauffman students are also more likely to be black and less 
likely to be Hispanic, are less likely to receive baseline test accommodations, and had higher 4th- 
grade attendance rates than students enrolled in Kansas City district schools. The same directional 
trends exist for Kauffman students relative to other Kansas City charter school students, but the 
differences are less pronounced and less likely to be statistically significant. Kauffman students 
were generally similar to other Kansas City students with respect to free or reduced-price lunch 
status and disability status, though there were some significant differences across cohorts. 


Table A.2. Baseline 4th-grade average characteristics of Kauffman students 
and other Kansas City public school students: Cohort I 8th graders 



Kauffman 

School 

All Kansas 
City public 
schools 

Kansas City 
district 
schools 

Kansas City 
charter 
schools 

4th-grade MAP mathematics scaled score 

637 

629* 

628* 

630 

4th-grade MAP ELA scaled score 

651 

639** 

635** 

645 

Free or reduced-price lunch 

0.86 

0.89 

0.92 

0.85 

Black 

0.81 

0.63** 

0.58** 

0.69* 

Hispanic 

0.13 

0.25** 

0.30** 

0.18 

Male 

0.49 

0.48 

0.51 

0.45 

Disabled 

0.07 

0.09 

0.09 

0.09 

Any baseline test accommodation 

0.08 

0.21** 

0.28** 

0.12 

4th-grade attendance rate 

0.95 

0.94** 

0.93** 

0.95 

4th-grade ever suspended 

0.18 

0.17 

0.15 

0.19 

Sample size 

88 

1,455 

832 

629 


‘Significantly different from Kauffman students at the 5 percent level. 
“Significantly different from Kauffman students at the 1 percent level. 
ELA = English language arts. 


4 The statewide average 4th-grade MAP scaled score was 648 in mathematics and 661 in ELA. The statewide 
standard deviation of 4th-grade MAP scores was 34 in mathematics and 38 in ELA. These numbers are calculated 
by averaging the year-specific means and standard deviations from 2010-201 1 through 2013-2014. 
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Table A.3. Baseline 4th-grade average characteristics of Kauffman students 
and other Kansas City public school students: Cohort II 7th graders 



Kauffman 

School 

All Kansas 
City public 
schools 

Kansas City 
district 
schools 

Kansas City 
charter 
schools 

4th-grade mathematics scaled score 

643 

633** 

630** 

636* 

4th-grade ELA scaled score 

651 

640** 

636** 

646 

Free or reduced-price lunch 

0.80 

0.89* 

0.94** 

0.84 

Black 

0.77 

0.62** 

0.56** 

0.69 

Hispanic 

0.08 

0.25** 

0.32** 

0.16* 

Male 

0.52 

0.47 

0.50 

0.46 

Disabled 

0.12 

0.08 

0.08 

0.09 

Any baseline test accommodation 

0.14 

0.24* 

032** 

0.14 

4th-grade attendance rate 

0.97 

0.95** 

0.95** 

0.95** 

4th-grade ever suspended 

0.11 

0.19* 

0.19* 

0.20* 

Sample size 

83 

1,472 

820 

665 

*Significantly different from Kauffman students at the 5 percent level. 

**Significantly different from Kauffman students at the 1 percent level. 


ELA = English language arts. 
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Table A.4. Baseline 4th-grade average characteristics of Kauffman students 
and other Kansas City public school students: Cohort III 6th graders 


Kauffman 

School 

All Kansas City 
public schools 

Kansas City Kansas City 

district schools charter schools 

4th-grade mathematics scaled score 

635 

632 

628** 

638 

4th-grade ELA scaled score 

653 

640** 

633** 

650 

Free or reduced-price lunch 

0.89 

0.90 

0.94 

0.86 

Black 

0.80 

0.58** 

0.53** 

0.64** 

Hispanic 

0.11 

0.27** 

0.32** 

0.22** 

Male 

0.44 

0.50 

0.53* 

0.46 

Disabled 

0.06 

0.10 

0.11* 

0.09 

Any baseline test accommodation 

0.12 

0.29** 

0.38** 

0.17 

4th-grade attendance rate 

0.96 

0.95** 

0.95** 

0.95 

4th-grade ever suspended 

0.15 

0.15 

0.16 

0.12 

Sample size 

179 

1,592 

965 

645 


*Significantly different from Kauffman students at the 5 percent level. 
**Significantly different from Kauffman students at the 1 percent level. 
ELA = English language arts. 


Table A.5. Baseline 4th-grade average characteristics of Kauffman students 


and other Kansas City public school students: Cohort IV 5th graders 



Kauffman 

School 

All Kansas City 
public schools 

Kansas City 
district schools 

Kansas City 
charter schools 

4th-grade mathematics scaled score 

636 

627** 

624** 

632 

4th-grade ELA scaled score 

650 

638** 

631** 

647 

Free or reduced-price lunch 

0.94 

0.88** 

0.92 

0.82** 

Black 

0.87 

0.54** 

0.50** 

0.59** 

Hispanic 

0.07 

0.30** 

0.35** 

0.23** 

Male 

0.45 

0.49 

0.51 

0.45 

Disabled 

0.08 

0.13* 

0.14** 

0.10 

Any baseline test accommodation 

0.07 

0.35** 

0.46** 

0.20** 

4th-grade attendance rate 

0.96 

0.95* 

0.95** 

0.95 

4th-grade ever suspended 

0.17 

0.16 

0.15 

0.17 

Sample size 

187 

1,585 

915 

691 

*Significantly different from Kauffman students at the 5 percent level. 

**Significantly different from Kauffman students at the 1 percent level. 

ELA = English language arts. 
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3. Multiple imputation methodology 

We calculated impact estimates using a multiple imputation procedure with M = 10 imputed 
data sets. We imputed missing baseline outcome variable values separately by treatment or 
comparison status using a chained linear equations model that included all outcome variables and 
all student characteristic variables included in the final impact regressions. 

Students were excluded from the imputation model if they had missing data for all 3rd- or 
4th-grade MAP test scores or missing data for all outcome (5th-, 6th-, 7th-, or 8th-grade) MAP 
test scores. Missing values were imputed before propensity-score matching and regression 
analyses in each multiple imputation data set. 

In addition to imputing baseline test scores, we imputed mathematics scores for students 
who took the 8th-grade Algebra I end-of-course exam in place of the 8th-grade mathematics 
MAP exam. In the 8th-grade mathematics imputation we included 8th-grade ELA and science 
MAP test scores, 7th-grade ELA and mathematics test MAP scores, 8th-grade attendance and 
suspension data, and the same set of student baseline characteristic variables included in the 
other imputations. 

After collecting coefficient and standard error estimates from each of the 10 imputed data 
sets, we computed multiple imputation coefficients and standard errors using Rubin’s 
combination method (Rubin 1987). The multiple imputation beta (/? M ) coefficient is the average 
of the beta coefficient values in each imputed data set (/? m ); the multiple imputation standard 
error is the square root of the within-imputation coefficient variance ( Var w ) plus the between- 
imputation coefficient variance (V ar B ) inflated by a finite imputation correction multiplier: 


(1) SE m = 


M 


Var w + 


K) 


Var B = 



~ fa ) 2 

M - 1 


4. Propensity-score matching methodology 

We estimated a propensity score for each eligible treatment and comparison student in each 
multiple imputation data set using a stepwise logistic regression model. We used an entry 
criterion of (p < .20) to determine whether each variable would enter the final logistic regression 
model. (See Table A.6 for a list of the variables.) 
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Table A.6. List of potential covariates used for propensity-score matching 

4th-grade mathematics and English language arts MAP z-scores 

Second- and third-order polynomials of 4th-grade mathematics and English language arts MAP z-scores 
3rd-grade mathematics and English language arts MAP z-scores 
4th-grade attendance rate and ever-suspended variables 

Gender, race, individualized education plan, English language learner, free or reduced-price lunch, any baseline 
test accommodation 

Interactions of 4th-grade mathematics and English language arts MAP z-scores with gender, race, individualized 
education plan, English language learner, free or reduced-price lunch, any baseline test accommodation 

Interactions of race with gender and free or reduced-price lunch 

Indicators for imputed 3rd- and 4th-grade mathematics and English language arts MAP z-score variables 
Indicator for imputed 4th-grade attendance rate or ever-suspended variables 


After generating propensity scores for each Kauffman student and eligible comparison 
student, we selected a matched comparison group by finding comparison students with 
propensity scores within a given threshold, or radius, from each Kauffman student’s propensity 
score. Comparison students were sampled with replacement, which means that each comparison 
student could be matched to multiple Kauffman students. To limit the number of possible 
comparison students, we specified a minimum matching radius and maximum number of 
potential matched neighbors. Because district students differed more from Kauffman students on 
baseline characteristics relative to the other two groups, we made the matching radius larger for 
the district comparison group to prevent the sample sizes of Kauffman and matched comparison 
students from being too small. If there were no comparison students within the matching radius 
for a given treatment student, that student was excluded from the matched comparison impact 
analyses. Because each comparison student could be matched to multiple treatment students, we 
used a weighting scheme in which each treatment student had a weight of one and each 
comparison student had a weight representing the number of matching treatment students. Table 
A.7 gives summary matching information for each comparison group. 5 


5 


Johnson et al. (2016) contains the matching information for other grade/cohort combinations. 
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Table A.7. Matching information summary 


All Kansas City 

Kansas City 

Kansas City 


public schools 

district schools 

charter schools 

Cohort 1 8th graders 


Minimum matching radius 

0.0006 

0.0016 

0.0008 

Maximum number of matches 

20 

20 

20 

Number of Kauffman students 

88 

88 

88 

Mean number of Kauffman students matched 

85 

76 

80 

Mean number of comparison students 

663 

381 

262 

Mean matches per Kauffman student 

11.2 

7.4 

4.8 

Cohort II 7th graders 

Minimum matching radius 

0.0006 

0.0016 

0.0008 

Maximum number of matches 

20 

20 

20 

Number of Kauffman students 

83 

83 

83 

Mean number of Kauffman students matched 

73 

65 

68 

Mean number of comparison students 

507 

343 

223 

Mean matches per Kauffman student 

9.1 

7.3 

4.1 

Cohort III 6th graders 

Minimum matching radius 

0.0006 

0.0016 

0.0008 

Maximum number of matches 

20 

20 

20 

Number of Kauffman students 

179 

179 

179 

Mean number of Kauffman students matched 

160 

153 

144 

Mean number of comparison students 

864 

602 

313 

Mean matches per Kauffman student 

7.9 

6.9 

3.4 

Cohort IV 5th graders 

Minimum matching radius 

0.0006 

0.0016 

0.0008 

Maximum number of matches 

20 

20 

20 

Number of Kauffman students 

187 

187 

187 

Mean number of Kauffman students matched 

171 

158 

156 

Mean number of comparison students 

543 

361 

288 

Mean matches per Kauffman student 

4.6 

3.6 

3.9 


In Table A.8, we present summary statistics to show how well Kauffman students were 
matched to comparison students on baseline characteristics. On average, comparison students 
from each matched group were not significantly different from Kauffman students on any 
baseline characteristics used in the analysis. Note that the sample sizes in Table A.8 are smaller 
for both Kauffman and comparison students relative to those in tables A.2-A.5. This is because 
some Kauffman students differed enough from all comparison students such that no good match 
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for these students could be found. 6 The matched comparison analysis excludes these Kauffman 
students. (Appendix B.2 contains a sensitivity analysis in which these students are included.) 

Table A.8. Baseline 4th-grade average characteristics of matched 
comparison samples 


Kauffman 

All Kansas City 

Kansas City 

Kansas City 


School 

public schools 

district schools 

charter schools 

Cohort 1 8th graders 

4th-grade mathematics scaled 





score 

635 (32) 

636 (35) 

635 (36) 

636 (32) 

4th-grade ELA scaled score 

649 (34) 

650 (35) 

646 (33) 

651 (35) 

Free or reduced-price lunch 

0.86 (0.35) 

0.85 (0.36) 

0.88 (0.33) 

0.87 (0.34) 

Black 

0.81 (0.40) 

0.79 (0.40) 

0.78 (0.42) 

0.80 (0.40) 

Hispanic 

0.12 (0.33) 

0.12 (0.33) 

0.13 (0.34) 

0.11 (0.31) 

Male 

0.47 (0.50) 

0.45 (0.50) 

0.45 (0.50) 

0.43 (0.50) 

Disabled 

0.07 (0.26) 

0.07 (0.25) 

0.07 (0.26) 

0.07 (0.25) 

Any prior test accommodation 

0.08 (0.28) 

0.07 (0.26) 

0.10 (0.29) 

0.07 (0.25) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

4th-grade ever suspended 

0.18 (0.39) 

0.17 (0.37) 

0.18 (0.38) 

0.18 (0.39) 

Sample size 

85 

663 

381 

262 

Cohort II 7th graders 

4th-grade mathematics scaled 





score 

643 (28) 

643 (28) 

641 (28) 

643 (28) 

4th-grade ELA scaled score 

652 (31) 

651 (32) 

648 (32) 

651 (32) 

Free or reduced-price lunch 

0.85 (0.36) 

0.86 (0.35) 

0.90 (0.29) 

0.81 (0.39) 

Black 

0.77 (0.43) 

0.74 (0.44) 

0.71 (0.46) 

0.76 (0.43) 

Hispanic 

0.10 (0.30) 

0.10 (0.30) 

0.11 (0.31) 

0.08 (0.27) 

Male 

0.51 (0.50) 

0.48 (0.50) 

0.50 (0.50) 

0.47 (0.50) 

Disabled 

0.10 (0.31) 

0.09 (0.28) 

0.12 (0.32) 

0.07 (0.26) 

Any prior test accommodation 

0.12 (0.33) 

0.14 (0.35) 

0.19 (0.39) 

0.10 (0.29) 

4th-grade attendance rate 

0.96 (0.03) 

0.96 (0.04) 

0.96 (0.04) 

0.96 (0.04) 

4th-grade ever suspended 

0.12 (0.33) 

0.13 (0.33) 

0.16 (0.37) 

0.13 (0.33) 

Sample size 

73 

507 

343 

223 


6 The composition of Kaufman students included in each matched comparison group analysis differs slightly 
among the separate analyses based on each comparison group. In Table A. 8, we report averages for Kauffman 
students included in the main analysis in which the comparison group includes all Kansas City public schools. 
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Kauffman 

School 

All Kansas City 
public schools 

Kansas City 
district schools 

Kansas City 
charter schools 

Cohort III 6th graders 

4th-grade mathematics scaled 





score 

634 (32) 

635 (31) 

633 (33) 

634 (31) 

4th-grade ELA scaled score 

648 (33) 

649 (34) 

647 (36) 

647 (32) 

Free or reduced-price lunch 

0.88 (0.32) 

0.90 (0.30) 

0.90 (0.29) 

0.92 (0.27) 

Black 

0.78 (0.42) 

0.77 (0.42) 

0.76 (0.43) 

0.78 (0.42) 

Hispanic 

0.12 (0.33) 

0.14 (0.35) 

0.15 (0.36) 

0.14 (0.35) 

Male 

0.45 (0.50) 

0.46 (0.50) 

0.46 (0.50) 

0.42 (0.49) 

Disabled 

0.07 (0.25) 

0.06 (0.23) 

0.07 (0.26) 

0.08 (0.27) 

Any prior test accommodation 

0.13 (0.34) 

0.14 (0.35) 

0.14 (0.35) 

0.13 (0.34) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

4th-grade ever suspended 

0.15 (0.36) 

0.14 (0.35) 

0.14 (0.35) 

0.14 (0.35) 

Sample size 

160 

864 

602 

313 

Cohort IV 5th graders 

4th-grade mathematics scaled 





score 

634 (27) 

633 (28) 

634 (28) 

633 (26) 

4th-grade ELA scaled score 

647 (32) 

646 (31) 

646 (31) 

649 (31) 

Free or reduced-price lunch 

0.94 (0.25) 

0.94 (0.23) 

0.92 (0.27) 

0.95 (0.22) 

Black 

0.88 (0.33) 

0.88 (0.33) 

0.86 (0.35) 

0.87 (0.33) 

Hispanic 

0.08 (0.26) 

0.07 (0.26) 

0.08 (0.26) 

0.08 (0.27) 

Male 

0.43 (0.50) 

0.40 (0.49) 

0.43 (0.50) 

0.38 (0.49) 

Disabled 

0.07 (0.25) 

0.07 (0.25) 

0.06 (0.24) 

0.07 (0.24) 

Any prior test accommodation 

0.08 (0.27) 

0.07 (0.26) 

0.08 (0.27) 

0.08 (0.27) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

0.95 (0.04) 

4th-grade ever suspended 

0.18 (0.38) 

0.18 (0.39) 

0.13 (0.33) 

0.15 (0.36) 

Sample size 

171 

543 

361 

288 


Notes: The Kauffman characteristics and sample size represent the total number of Kauffman students matched to 

the full comparison group of students from all Kansas City public schools. Standard deviations are 
displayed in parentheses next to the averages in this table. No differences between averages for Kauffman 
students and comparison group students are significantly different from zero. 

ELA = English language arts. 
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1. Attrition-adjusted results 

Between 7 and 10 percent of cohort I through IV 5th graders who were enrolled in the 
Kauffman School at the beginning of the year left the Kauffman School before taking the 5th- 
grade Missouri Assessment Program (MAP) exams. Most of them enrolled in other schools in 
Missouri and took the MAP exams at the end of the year, so we are able to track their 
achievement and include them in the analyses. Including these students means that the estimated 
effect sizes are interpretable as the additional achievement growth a student enrolling in the 
Kauffman School is expected to attain, accounting for the chance that this student might drop out 
during the school year. Although this is an informative number to calculate and is comparable to 
estimates reported in other charter school evaluations, it could also be of interest to estimate the 
effect of the Kauffman School on students who remain enrolled in the Kauffman School. 

An estimate of the impact of the Kauffman School on the achievement of students who 
remained enrolled can be calculated by applying an adjustment for attrition known as a Bloom 
adjustment (Bloom 1984), which is calculated by dividing the impact estimates by the fraction of 
students who remained enrolled in the Kauffman School for the entire year. This adjustment is 
made under the assumption that the end-of-year outcomes for students who withdrew from the 
Kauffman School are unaffected by their enrollment in the Kauffman School. This assumption is 
unlikely to be true for students who withdrew later in the year, which means that the attrition- 
adjusted effect sizes are likely to be biased upward. The attrition-adjusted one-year effect size 
estimates are displayed in Table B.l. We do not present attrition-adjusted results for the two-, 
three-, or four- year impact estimates, because many of the students in that analysis spent their 
entire 5th-grade year enrolled in the Kauffman School before dropping out in 6th, 7th, or 8th 
grade. For these students, the assumption of zero impact of the Kauffman School made when 
calculating the attrition-adjusted results would clearly not hold. 


Table B.l. Attrition-adjusted impact of Kauffman School on MAP test scores 
(citywide comparison group) 



Attrition-adjusted results 

Benchmark results 

One-year impact estimates 

5th-grade mathematics effect size 

0.24** 

0.22** 


(0.03) 

(0.03) 

5th-grade ELA effect size 

0.25** 

0.23** 


(0.03) 

(0.03) 

5th-grade science effect size 

0.49** 

0.46** 


(0.04) 

(0.04) 

Sample size 

2,956 

2,956 


Notes: This table presents the attrition-adjusted average one-year impact estimates in effect size units. Standard 
errors are displayed in parentheses below each impact estimate. The sample size represents the total 
number of Kauffman students and matched comparison students entering each analysis. 


“Significantly different from zero at the 1 percent level. 

ELA = English language arts. 

As shown in Table B.l, the one-year impact estimates are larger but overall very similar in 
all three subjects when adjusted for attrition: 0.25 in mathematics, 0.24 in ELA, and 0.49 in 
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science. The significance level for all effect size estimates from the attrition-adjusted results 
remains unchanged from the main results. 1 The attrition-adjusted attendance and suspension 
impact estimates are displayed in Table B.2. They are also slightly larger in absolute value, but 
broadly similar to the main results. 


Table B.2. Attrition-adjusted impact of Kauffman School on attendance and 
suspensions (citywide comparison group) 



Attrition-adjusted results 

Benchmark results 

Cohort IV 5th graders 

Impact on attendance rate (%) 

1 .28** 

(0.43) 

1.13** 

(0.40) 

Impact on probability of being suspended (%) 

14.8** 

(4.8) 

13.1** 

(4.4) 

Sample size 

714 

714 

Cohort III 5th graders 

Impact on attendance rate (%) 

0.64 

0.57 


(0.43) 

(0.39) 

Impact on probability of being suspended (%) 

31.3** 

27.9** 


(4.0) 

(3.6) 

Sample size 

948 

948 

Cohort II 5th graders 

Impact on attendance rate (%) 

0.91 

(0.49) 

0.84 

(0.45) 

Impact on probability of being suspended (%) 

1.6 

(4.9) 

1.5 

(4.5) 

Sample size 

617 

617 

Cohort 1 5th graders 

Impact on attendance rate (%) 

-0.93 

(0.53) 

-0.83 

(0.48) 

Impact on probability of being suspended (%) 

14.9* 

(5.8) 

13.4* 

(5.2) 

Sample size 

677 

677 


Notes: This table reports the attrition-adjusted estimated impact of the Kauffman School on attendance and 

suspensions. The suspension results are marginal effects from logit models in which the outcome variable 
is an indicator for receiving a suspension during the year. Standard errors are displayed in parentheses 
below each impact estimate. The sample size represents the total number of Kauffman students and 
matched comparison students entering each analysis. 

*Significantly different from zero at the 5 percent level. 

**Significantly different from zero at the 1 percent level. 


1 Because the standard errors are adjusted along with the impact estimates, the statistical significance of the 
results will not change after the attrition adjustment. 
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2. Sensitivity of results to comparison group students 

To examine the sensitivity of the results to the choice of comparison group, we re-estimated 
the models including all students in Kansas City in relevant grade levels in the comparison 
group, even those whose baseline characteristics differed from those of Kauffman students. 
Rather than match students on baseline characteristics, this method relies exclusively on statistical 
controls for baseline characteristics. The results are displayed in Table B.3. The results using all 
Kauffman and comparison students are nearly identical to the results based on the matched 
comparison group. This indicates that including the baseline control variables in a regression 
framework performed well in reducing bias that might result from the inclusion of comparison 
students who differed from Kauffman students in terms of baseline characteristics. In other 
charter school evaluations, regression results based on all comparison students have been shown 
to closely approximate results based on matched comparison groups (Tuttle et al. 2013). 


Table B.3. Impact of Kauffman School on MAP test scores using full Kansas 
City comparison group 



Full Kansas City 
comparison results 

Benchmark matched comparison 
results 

One-year impacts 

5th-grade mathematics effect size 

0.19** 

0.22** 


(0.03) 

(0.03) 

5th-grade ELA effect size 

0.22** 

0.23** 


(0.03) 

(0.03) 

5th-grade science effect size 

0.45** 

0.46** 


(0.03) 

(0.04) 

Sample size 

7,211 

2,956 

Two-year impacts 

6th-grade mathematics effect size 

0.35** 

0.35** 


(0.03) 

(0.04) 

6th-grade ELA effect size 

0.19** 

0.18** 


(0.03) 

(0.04) 

Sample size 

5,100 

2,205 

Three-year impacts 

7th-grade mathematics effect size 

0.67** 

0.68** 


(0.04) 

(0.05) 

7th-grade ELA effect size 

0.52** 

0.52** 


(0.05) 

(0.06) 

Sample size 

3,155 

1,115 
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Full Kansas City 
comparison results 

Benchmark matched comparison 
results 

Four-year impacts 

8th-grade mathematics effect size 

0.95** 

0.96** 


(0.10) 

(0.10) 

8th-grade ELA effect size 

0.51** 

0.53** 


(0.07) 

(0.08) 

8th-grade science effect size 

0.65** 

0.66** 


(0.09) 

(0.10) 


Sample size 1,543 748 


Notes: This table displays impact estimates in effect size units for the full Kansas City comparison sample 

alongside the benchmark estimates from the matched comparison sample. The first section of this table 
presents the average one-year impact estimates for cohort I, II, III, and IV 5th graders. The second section 
presents the average two-year impact estimates for cohort 1,11, and III 6th graders. The third section 
presents three-year impact estimates for cohort I and II 7th graders. The fourth section presents four-year 
impact estimates for cohort I 8th graders. Standard errors are displayed in parentheses below each impact 
estimate. The sample size represents the total number of Kauffman students and comparison students 
entering each analysis. 

**Significantly different from zero at the 1 percent level. 

ELA = English language arts. 


3. Sensitivity of results to exclusion of grade repeaters 

A small percentage of Kauffman’s cohort I, II, and III students (3.4 percent) repeated 5th 
grade in 2012-13, 2013-14, or 2014-15. This is slightly higher than the percentage of 5th-grade 
repeaters in Kansas City (1.1 percent). No cohort I or II students repeated 6th grade in 2013-14 
or 2014-15, though 1.4 percent of other Kansas City students did. In 2014-15, 2.9 percent of 
Kauffman’s cohort I repeated the 7th grade as did 2.6 percent of other Kansas City students. 
When a student repeats a grade, it creates a missing-data problem for the analysis because that 
student no longer takes the same outcome assessment as the rest of the students in his or her 
original cohort. Excluding repeaters from the analyses might introduce bias in the two- or three- 
year impact estimates for the Kauffman School, because repeater students are likely to struggle 
in terms of achievement growth. We therefore included repeaters in our main analyses. We 
follow the method used in Tuttle et al. (2013) for dealing with missing outcome scores for 
repeaters, which involves assuming that the relative rank in the district test score distribution 
does not change after the first time the repeater completed his or her previous grade. For 
example, students who repeat 5th grade are included along with other students from their same 
cohort in the two-, three-, and four-year impact estimates, with the z-scores of the repeater 
students fixed at their end-of-5th-grade values. 2 Cohort I students who repeated any grade were 
missing 8th-grade science scores, so we used their 5th-grade science z-scores for our main 
analysis. Because we have shown that the Kauffman School has positive impacts on student 


2 

In previous reports we performed this step of substituting of prior outcome z-scores for repeaters before 
imputing missing baseline test scores. To improve the accuracy of the imputation procedure for the year 4 impact 
estimates, we performed this substitution step after the imputation of missing baseline scores. This update had little 
effect on the impact estimates; the majority remained the same and those that changed moved by only 0.01 standard 
deviations. Because the results were very similar with and without this change, we did not re-calculate the impact 
estimates from previous years. 
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achievement, the assumption about the test scores of repeaters will likely bias the two-, three-, 
and four- year impact estimates downward, because we are assuming that the Kauffman School 
has no effect on repeaters during their subsequent years enrolled. 

In Table B.4, we present the results from our two- through four- year impact estimates when 
grade repeaters are excluded from the analysis. These effect sizes will likely provide an upper 
bound on the estimated effect size for the Kauffman School, because it retains students at a 
higher rate in 5th grade compared with other Kansas City schools. Nonetheless, the results do not 
differ substantially from our benchmark results. 


Table B.4. Impact of Kauffman School on MAP test scores (citywide 
comparison group), excluding grade repeaters 



Results excluding repeaters 

Benchmark results 
including repeaters 

Two-year impact estimates 

Mathematics effect size 

0.35** 

0.35** 


(0.04) 

(0.04) 

ELA effect size 

0.19** 

0.18** 


(0.04) 

(0.04) 

Sample size 

2,083 

2,205 

Three-year impact estimates 

Mathematics effect size 

0.70** 

0.68** 


(0.06) 

(0.05) 

ELA effect size 

0.55** 

0.52** 


(0.06) 

(0.06) 

Sample size 

1,049 

1,115 

Four-year impact estimates 

Mathematics effect size 

1 .03“ 

0.96“ 


(0.11) 

(0.10) 

ELA effect size 

0.56** 

0.53** 


(0.08) 

(0.08) 

Science effect size 

0.70** 

0.66** 


(0.11) 

(0.10) 

Sample size 

676 

748 


Notes: This table displays impact estimates in effect size units. The first column presents the two-year impact 

estimate for 6th graders, the three-year impact estimate for 7th graders, and the four-year impact estimate 
for 8th graders when students who repeat 5th, 6th, or 7th grade are excluded from the analysis. Standard 
errors are displayed in parentheses below each impact estimate. The sample size represents the total 
number of Kauffman students and matched comparison students entering each analysis. 

‘‘Significantly different from zero at the 1 percent level. 

ELA = English language arts. 

4. Sensitivity of results to imputation procedure 

The use of 10 imputed data sets in our multiple imputation procedure should be sufficient to 
prevent randomness in the imputed values from influencing the results. However, to test the 
sensitivity of the results to the use of imputed data, we re-estimated the regression models 
excluding observations that had any imputed baseline values. For the four-year impact analysis. 
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we also excluded observations with imputed 8th grade mathematics MAP test scores. Grade 
repeaters were excluded from this sensitivity analysis, so that all impact estimates are based only 
on observed data. In this section, we report detailed matching information, baseline equivalence 
results, and MAP impact estimates when no imputed data are used. 

The matching information for each cohort and comparison group is displayed in Table B.5. 
We used the same matching radius and maximum number of matches per Kauffman student as in 
the main results (Table A.7). The primary difference in the match statistics in Table B.5 
compared with Table A.7 is that because students with imputed data are excluded from the 
sample, there are fewer Kauffman students and fewer comparison students. 


Table B.5. Matching information summary for results using no imputed data 



No imputed values 

Including imputed values 

Cohort 1 8th graders 

Minimum matching radius 

0.0006 

0.0006 

Maximum number of matches 

20 

20 

Number of Kauffman students 

75 

88 

Number of Kauffman students matched 

73 

85 

Number of comparison students 

485 

663 

Mean matches per Kauffman student 

9.7 

11.2 

Cohort II 7th graders 

Minimum matching radius 

0.0006 

0.0006 

Maximum number of matches 

20 

20 

Number of Kauffman students 

77 

83 

Number of Kauffman students matched 

69 

73 

Number of comparison students 

478 

507 

Mean matches per Kauffman student 

8.8 

9.1 

Cohort III 6th graders 

Minimum matching radius 

0.0006 

0.0006 

Maximum number of matches 

20 

20 

Number of Kauffman students 

161 

179 

Number of Kauffman students matched 

140 

160 

Number of comparison students 

703 

864 

Mean matches per Kauffman student 

7.0 

7.9 

Cohort IV 5th graders 

Minimum matching radius 

0.0006 

0.0006 

Maximum number of matches 

20 

20 

Number of Kauffman students 

179 

187 

Number of Kauffman students matched 

159 

171 

Number of comparison students 

478 

543 

Mean matches per Kauffman student 

4.4 

4.6 


The average baseline characteristics of Kauffman and comparison students when no imputed 
data are included are displayed in Table B.6. As with the baseline statistics displayed for the 
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main analysis results (Table A.8), there are no statistically significant differences between the 
averages for Kauffman and comparison students on any of the baseline characteristics we 
examined. Appendix B of Johnson et al. (2016) provides the matching information and a 
comparison of the baseline characteristics for other cohort and grade combinations. 3 

Table B.6. Baseline 4th-grade average characteristics of matched 
comparison sample: Results using no imputed data 


Kauffman School 

All Kansas City public schools 

Cohort 1 8th graders 

4th-grade mathematics scaled score 

637 (32) 

638 (35) 

4th-grade ELA scaled score 

654 (33) 

656 (33) 

Free or reduced-price lunch 

0.85 (0.36) 

0.84 (0.37) 

Black 

0.81 (0.40) 

0.81 (0.39) 

Hispanic 

0.12 (0.33) 

0.12 (0.32) 

Male 

0.47 (0.50) 

0.42 (0.49) 

Disabled 

0.07 (0.25) 

0.09 (0.29) 

Any prior test accommodation 

0.10 (0.30) 

0.13 (0.33) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

4th-grade ever suspended 

0.19 (0.40) 

0.17 (0.37) 

Sample size 

73 

485 

Cohort II 7th graders 

4th-grade mathematics scaled score 

642 (27) 

645 (26) 

4th-grade ELA scaled score 

652 (32) 

656 (32) 

Free or reduced-price lunch 

0.83 (0.38) 

0.87 (0.33) 

Black 

0.77 (0.43) 

0.76 (0.43) 

Hispanic 

0.10 (0.30) 

0.10 (0.30) 

Male 

0.49 (0.50) 

0.47 (0.50) 

Disabled 

0.12 (0.32) 

0.07 (0.25) 

Any prior test accommodation 

0.13 (0.34) 

0.13 (0.34) 

4th-grade attendance rate 

0.96 (0.03) 

0.96 (0.04) 

4th-grade ever suspended 

0.13 (0.34) 

0.12 (0.33) 

Sample size 

69 

478 

Students who repeated a grade were inadvertently included in this robustness check in Johnson et al. (2016). 
For results presented in this section, which average 2014-2015 effect size estimates with those from previous years, 
we updated the 2013-2014 results to correctly exclude repeaters from the analysis. Revised results for this 
robustness check from Johnson et al. (2016) are available upon request. 
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Kauffman School 

All Kansas City public schools 

Cohort III 6th graders 

4th-grade mathematics scaled score 

634 (29) 

637 (30) 

4th-grade ELA scaled score 

648 (32) 

652 (32) 

Free or reduced-price lunch 

0.88 (0.33) 

0.88 (0.33) 

Black 

0.77 (0.42) 

0.75 (0.43) 

Hispanic 

0.13 (0.34) 

0.15 (0.36) 

Male 

0.43 (0.50) 

0.44 (0.50) 

Disabled 

0.07 (0.26) 

0.06 (0.23) 

Any prior test accommodation 

0.14 (0.34) 

0.14 (0.35) 

4th-grade attendance rate 

0.95 (0.04) 

0.95 (0.04) 

4th-grade ever suspended 

0.15 (0.36) 

0.13 (0.34) 

Sample size 

140 

703 

Cohort IV 5th graders 

4th-grade mathematics scaled score 

633 (27) 

634 (28) 

4th-grade ELA scaled score 

646 (33) 

647 (29) 

Free or reduced-price lunch 

0.94 (0.23) 

0.94 (0.23) 

Black 

0.89 (0.31) 

0.88 (0.32) 

Hispanic 

0.08 (0.26) 

0.06 (0.24) 

Male 

0.46 (0.50) 

0.47 (0.50) 

Disabled 

0.07 (0.25) 

0.08 (0.27) 

Any prior test accommodation 

0.08 (0.27) 

0.08 (0.26) 

4th-grade attendance rate 

0.95 (0.04) 

0.96 (0.04) 

4th-grade ever suspended 

0.18 (0.39) 

0.17 (0.38) 

Sample size 

159 

478 

Notes: Standard deviations are displayed in parentheses next to the averages in this table. No differences 

between averages for Kauffman students and comparison group students are significantly different from 

zero. 



ELA = English language arts. 



The test score impact estimates based on students with nonmissing data are displayed in 
Table B.7. The results are broadly similar to the main results shown in Table III. 1 . The impact 
estimates are similar in magnitude, and the statistical significance is the same as for the main 

results. 
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Table B.7. Impact of Kauffman School on MAP test scores (citywide 
comparison group) using no imputed data 



Results using 
no imputed data 

Benchmark results 
using imputed data 

One-year impact estimates 

5th-grade mathematics effect size 

0.20** (0.03) 

0.22** (0.03) 

5th-grade ELA effect size 

0.24** (0.03) 

0.23** (0.03) 

5th-grade science effect size 

0.46** (0.03) 

0.46** (0.04) 

Sample size 

2,568 

2,956 

Two-year impact estimates 

6th-grade mathematics effect size 

0.37** (0.04) 

0.35** (0.04) 

6th-grade ELA effect size 

0.24** (0.04) 

0.18** (0.04) 

Sample size 

1,816 

2,205 

Three-year impact estimates 

7th-grade mathematics effect size 

0.69** (0.05) 

0.68** (0.05) 

7th-grade ELA effect size 

0.52** (0.06) 

0.52** (0.06) 

Sample size 

967 

1,115 

Four-year impact estimates 

8th-grade mathematics effect size 

1.16** (0.10) 

0.96** (0.10) 

8th-grade ELA effect size 

0.58** (0.08) 

0.53** (0.08) 

8th-grade science effect size 

0.73** (0.11) 

0.66** (0.10) 

Sample size 

558 

748 


Notes: This table displays impact estimates in effect size units. The first section of this table presents the average 
one-year impact estimates for cohort I, II, III, and IV 5th graders. The second section presents the average 
two-year impact estimates for cohort I, II, and III 6th graders. The third section presents the three-year 
impact estimates for cohort I and II 7th graders. The fourth section presents the four-year impact estimates 
for cohort I 8th graders. The first data column includes only students with nonmissing data in the analysis 
sample. Standard errors are displayed in parentheses below each impact estimate. The sample size 
represents the total number of Kauffman and matched comparison students entering each analysis. 

**Significantly different from zero at the 1 percent level. 

ELA = English language arts. 

5. Alternative comparison groups for attendance and suspension impact 
estimates 

We report in Table B.8 the attendance and suspension results when the two alternative 
comparison groups are used. The magnitude of the impact on suspensions is smaller when the 
charter school comparison group is used relative to the district school comparison group. 
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Table B.8. Alternate estimates of impact of Kauffman School on attendance 


and suspensions (district/charter comparisons) 



Impact on attendance 
rate (%) 

Impact on probability 
of being suspended 

(%) 

Sample size 

Kansas City district schools 

Cohort 1 8th graders 

0.55 (0.77) 

14.8 (6.1) 

457 

Cohort II 7th graders 

1.35 (0.90) 

5.33 (6.7) 

408 

Cohort III 6th graders 

1.12* (0.45) 

6.76 (4.4) 

755 

Cohort IV 5th graders 

1 .07* (0.46) 

16.9** (4.1) 

519 

Average across grades 

1.04** (0.29) 

11.4** (2.5) 

2,138 

Kansas City charter schools 

Cohort 1 8th graders 

0.08 (0.72) 

11.9 (6.9) 

343 

Cohort II 7th graders 

0.17 (0.78) 

4.83 (7.4) 

291 

Cohort III 6th graders 

0.82 (0.46) 

-1.03 (5.0) 

457 

Cohort IV 5th graders 

1.69** (0.45) 

8.95 (4.7) 

443 

Average across grades 

0.89** (0.28) 

5.64* (2.8) 

1,534 


Notes: This table reports the estimated impact of the Kauffman School on attendance and suspensions. The 

suspension results are marginal effects from logit models in which the outcome variable is an indicator for 
receiving a suspension during the year. Standard errors are displayed in parentheses below each impact 
estimate. The sample size represents the total number of Kauffman students and matched comparison 
students entering each analysis. The bottom row in each section displays the average across grades, 
weighted by the number of Kauffman students in each grade. 

*Significantly different from zero at the 5 percent level. 

**Significantly different from zero at the 1 percent level. 
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